pvops netconsole

DISCUSSION Brad Plant 3 participants 9 messages Feb 5, 2010 – Feb 16, 2010

Subsystems (4)

Network drivers x86 Xen hypervisor interface Netconsole

lore

Share & tools

lore

Brad Plant

author

2010-02-05 00:15 UTC #0 lore raw

Hi,

I tried asking this question on xen-users, but didn't get a response so I'm now asking on xen-devel instead....

Is netconsole supposed to work under pvops? When I try to enable it I get the messages below. I'm wondering if netconsole just doesn't work at all with pvops or whether it's dependent on the NIC used by the dom0.

Initialising Xen virtual ethernet driver.
blkfront: xvda1: barriers enabled
netconsole: local port 11111
netconsole: local IP 10.20.40.2
netconsole: interface eth1
netconsole: remote port 9514
netconsole: remote IP 10.20.50.2
netconsole: remote ethernet address 00:16:3e:00:50:02
netconsole: eth1 doesn't support polling, aborting.
netconsole: cleaning up

Cheers,

Brad

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Konrad Rzeszutek Wilk

2010-02-05 16:34 UTC #1 lore raw

Quoted 5 lines · Brad Plant

Hi,
I tried asking this question on xen-users, but didn't get a response so I'm now asking on xen-devel instead....
Is netconsole supposed to work under pvops? When I try to enable it I get the messages below. I'm wondering if netconsole just doesn't work at all with pvops or whether it's dependent on the NIC used by the dom0.

Hmmm, I _think_ I had it working at some point, but this was with dom0, not with domUs, which is what you are looking to be doing?

Quoted 1 line

Initialising Xen virtual ethernet driver.

Brad Plant

author

2010-02-05 20:21 UTC #2 lore raw

Quoted 4 lines · On Fri, 5 Feb 2010 11:34:30 -0500 Konrad Rzeszutek Wilk…

Quoted 1 line
Is netconsole supposed to work under pvops? When I try to enable it I get the messages below. I'm wondering if netconsole just doesn't work at all with pvops or whether it's dependent on the NIC used by the dom0.
Hmmm, I _think_ I had it working at some point, but this was with dom0, not with domUs, which is what you are looking to be doing?

That's correct - I'm trying to use netconsole on a domU.

Cheers,

Brad

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Jeremy Fitzhardinge

2010-02-12 23:09 UTC #3 lore raw

In reply to Brad Plant

Quoted 11 lines · Brad Plant

Initialising Xen virtual ethernet driver.
blkfront: xvda1: barriers enabled
netconsole: local port 11111
netconsole: local IP 10.20.40.2
netconsole: interface eth1
netconsole: remote port 9514
netconsole: remote IP 10.20.50.2
netconsole: remote ethernet address 00:16:3e:00:50:02
netconsole: eth1 doesn't support polling, aborting.
netconsole: cleaning up

The xen-netfront device doesn't support the polling interface that netconsole relies on. It probably wouldn't be too hard to add that capability if you want to. It's independent of the backend and/or the actual underlying hardware device.

Brad Plant

author

2010-02-13 00:05 UTC #4 lore raw

Quoted 13 lines · On Fri, 12 Feb 2010 15:09:12 -0800 Jeremy Fitzhardinge <[email protected]> wrote:

Quoted 11 lines · Brad Plant
Initialising Xen virtual ethernet driver.
blkfront: xvda1: barriers enabled
netconsole: local port 11111
netconsole: local IP 10.20.40.2
netconsole: interface eth1
netconsole: remote port 9514
netconsole: remote IP 10.20.50.2
netconsole: remote ethernet address 00:16:3e:00:50:02
netconsole: eth1 doesn't support polling, aborting.
netconsole: cleaning up

Quoted 4 lines

The xen-netfront device doesn't support the polling interface that netconsole relies on. It probably wouldn't be too hard to add that capability if you want to. It's independent of the backend and/or the actual underlying hardware device.

I like the sound of that. I'll give it a crack :-)

Cheers,

Brad

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Brad Plant

author

2010-02-14 02:28 UTC #5 lore raw

In reply to Jeremy Fitzhardinge

Quoted 13 lines · On Fri, 12 Feb 2010 15:09:12 -0800 Jeremy Fitzhardinge <[email protected]> wrote:

Quoted 11 lines · Brad Plant
Initialising Xen virtual ethernet driver.
blkfront: xvda1: barriers enabled
netconsole: local port 11111
netconsole: local IP 10.20.40.2
netconsole: interface eth1
netconsole: remote port 9514
netconsole: remote IP 10.20.50.2
netconsole: remote ethernet address 00:16:3e:00:50:02
netconsole: eth1 doesn't support polling, aborting.
netconsole: cleaning up

Quoted 4 lines

The xen-netfront device doesn't support the polling interface that netconsole relies on. It probably wouldn't be too hard to add that capability if you want to. It's independent of the backend and/or the actual underlying hardware device.

Ok, I've come up with something that "works". Patch against 2.6.32.7 is attached. While it works, I've got no idea if what I've done is the "right way". I used virtio_net.c as a guide.

I've got some issues that I'm stuck on and would like some guidance:

1. When the first message is sent over the net console, the WARN_ONCE() at net/core/netpoll.c:327 triggers. I'm not sure what to do about this.

2. When either netconsole is setup or the first message is sent, messages already logged to the main console get logged again. Maybe this will go away when the above is fixed?

3. Initially netconsole would not initialise because the interface didn't yet have a mac address as setup_netfront hadn't yet been called. I changed the module_init() in netconsole.c to late_initcall() if !CONFIG_MODULES && CONFIG_XEN. Is this allowed and sane?

Cheers,

Brad

drivers/net/netconsole.c +8 -0

diff -urpNX linux-2.6.32.7/Documentation/dontdiff linux-2.6.32.7/drivers/net/netconsole.c linux-2.6.32.7-xen-netconsole/drivers/net/netconsole.c
@@ -816,5 +816,13 @@ static void __exit cleanup_netconsole(vo
 	}
 }
 
+#ifndef CONFIG_MODULES
+#ifdef CONFIG_XEN
+late_initcall(init_netconsole);
+#else
 module_init(init_netconsole);
+#endif
+#else
+module_init(init_netconsole);
+#endif
 module_exit(cleanup_netconsole);

drivers/net/xen-netfront.c +12 -0

diff -urpNX linux-2.6.32.7/Documentation/dontdiff linux-2.6.32.7/drivers/net/xen-netfront.c linux-2.6.32.7-xen-netconsole/drivers/net/xen-netfront.c
@@ -989,6 +989,15 @@ err:
 	return work_done;
 }
 
+#ifdef CONFIG_NET_POLL_CONTROLLER
+static void xennet_netpoll(struct net_device *dev)
+{
+	struct netfront_info *np = netdev_priv(dev);
+
+	napi_schedule(&np->napi);
+}
+#endif
+
 static int xennet_change_mtu(struct net_device *dev, int mtu)
 {
 	int max = xennet_can_sg(dev) ? 65535 - ETH_HLEN : ETH_DATA_LEN;

@@ -1113,6 +1122,9 @@ static const struct net_device_ops xenne
 	.ndo_change_mtu	     = xennet_change_mtu,
 	.ndo_set_mac_address = eth_mac_addr,
 	.ndo_validate_addr   = eth_validate_addr,
+#ifdef CONFIG_NET_POLL_CONTROLLER
+	.ndo_poll_controller = xennet_netpoll,
+#endif
 };
 
 static struct net_device * __devinit xennet_create_dev(struct xenbus_device *dev)

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Jeremy Fitzhardinge

2010-02-14 03:56 UTC #6 lore raw

Quoted 23 lines · Brad Plant

Quoted 14 lines · On Fri, 12 Feb 2010 15:09:12 -0800 Jeremy Fitzhardinge<[email protected]> wrote:
Quoted 12 lines · Brad Plant
Initialising Xen virtual ethernet driver.
blkfront: xvda1: barriers enabled
netconsole: local port 11111
netconsole: local IP 10.20.40.2
netconsole: interface eth1
netconsole: remote port 9514
netconsole: remote IP 10.20.50.2
netconsole: remote ethernet address 00:16:3e:00:50:02
netconsole: eth1 doesn't support polling, aborting.
netconsole: cleaning up
Quoted 5 lines
The xen-netfront device doesn't support the polling interface that netconsole relies on. It probably wouldn't be too hard to add that capability if you want to. It's independent of the backend and/or the actual underlying hardware device.

Quoted 7 lines

Ok, I've come up with something that "works". Patch against 2.6.32.7 is attached. While it works, I've got no idea if what I've done is the "right way". I used virtio_net.c as a guide.
I've got some issues that I'm stuck on and would like some guidance:
1. When the first message is sent over the net console, the WARN_ONCE() at net/core/netpoll.c:327 triggers. I'm not sure what to do about this.

What's the backtrace in the warning?

I had a quick look at e1000's and it calls the interrupt handler; the netfront equivalent would be to call xennet_interrupt() with interrupts disabled. I'm not sure virtio_net is a good model because it doesn't seem to have any interrupt-related stuff in it; I guess that happens elsewhere in virtio.

Quoted 2 lines

2. When either netconsole is setup or the first message is sent, messages already logged to the main console get logged again. Maybe this will go away when the above is fixed?

Probably not. It sounds like a feature.

Quoted 2 lines

3. Initially netconsole would not initialise because the interface didn't yet have a mac address as setup_netfront hadn't yet been called. I changed the module_init() in netconsole.c to late_initcall() if !CONFIG_MODULES&& CONFIG_XEN. Is this allowed and sane?

Probably not. In practice every kernel will have modules and Xen enabled, and it will change netconsole for all users. I guess we probably need to find some way to init the MAC address earlier, or trigger netconsole init once the mac address is known.

Quoted 4 lines

Cheers,
Brad

Brad Plant

author

2010-02-14 04:56 UTC #7 lore raw

Quoted 3 lines · On Sat, 13 Feb 2010 19:56:51 -0800 Jeremy Fitzhardinge <[email protected]> wrote:

Quoted 1 line
1. When the first message is sent over the net console, the WARN_ONCE() at net/core/netpoll.c:327 triggers. I'm not sure what to do about this.
What's the backtrace in the warning?

------------[ cut here ]------------
WARNING: at net/core/netpoll.c:329 netpoll_send_skb+0x145/0x1db()
netpoll_send_skb(): eth1 enabled interrupts in poll (xennet_start_xmit+0x0/0x6bb)
Pid: 1, comm: swapper Not tainted 2.6.32.7 #36
Call Trace:
 [<ffffffff81039100>] warn_slowpath_common+0x72/0x8a
 [<ffffffff81039165>] warn_slowpath_fmt+0x3c/0x3e
 [<ffffffff8100dd7f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff8124e2a0>] ? xennet_start_xmit+0x0/0x6bb
 [<ffffffff81294e48>] netpoll_send_skb+0x145/0x1db
 [<ffffffff813604d4>] ? _spin_unlock_irqrestore+0x19/0x1c
 [<ffffffff812950dc>] netpoll_send_udp+0x1fe/0x20e
 [<ffffffff8124004d>] ? firmware_loading_store+0x120/0x1b5
 [<ffffffff8124f9d6>] write_msg+0x89/0xca
 [<ffffffff8124004d>] ? firmware_loading_store+0x120/0x1b5
 [<ffffffff81039281>] __call_console_drivers+0x67/0x79
 [<ffffffff810392ec>] _call_console_drivers+0x59/0x5d
 [<ffffffff81039613>] release_console_sem+0x120/0x1c7
 [<ffffffff81039e73>] register_console+0x22e/0x2ae
 [<ffffffff81503380>] init_netconsole+0x15d/0x1c8
 [<ffffffff81049033>] ? schedule_delayed_work+0x16/0x1b
 [<ffffffff81503223>] ? init_netconsole+0x0/0x1c8
 [<ffffffff8100a05f>] do_one_initcall+0x59/0x154
 [<ffffffff814e56a9>] kernel_init+0x152/0x1a8
 [<ffffffff810112aa>] child_rip+0xa/0x20
 [<ffffffff810104a1>] ? int_ret_from_sys_call+0x7/0x1b
 [<ffffffff81010c5d>] ? retint_restore_args+0x5/0x6
 [<ffffffff810112a0>] ? child_rip+0x0/0x20
---[ end trace 0f898deb8e1e2914 ]---

Quoted 5 lines

I had a quick look at e1000's and it calls the interrupt handler; the netfront equivalent would be to call xennet_interrupt() with interrupts disabled. I'm not sure virtio_net is a good model because it doesn't seem to have any interrupt-related stuff in it; I guess that happens elsewhere in virtio.

Actually, I tried something like this first because I first looked at pcnet32.c and e100.c:

disable_irq(dev->irq);
xennet_interrupt(dev->irq, dev);
enable_irq(dev->irq);

The same WARN_ONCE() was triggered.

Quoted 3 lines

Quoted 1 line
2. When either netconsole is setup or the first message is sent, messages already logged to the main console get logged again. Maybe this will go away when the above is fixed?
Probably not. It sounds like a feature.

Ok. This one was the least of my worries anyway.

Quoted 6 lines

Quoted 1 line
3. Initially netconsole would not initialise because the interface didn't yet have a mac address as setup_netfront hadn't yet been called. I changed the module_init() in netconsole.c to late_initcall() if !CONFIG_MODULES&& CONFIG_XEN. Is this allowed and sane?
Probably not. In practice every kernel will have modules and Xen enabled, and it will change netconsole for all users. I guess we probably need to find some way to init the MAC address earlier, or trigger netconsole init once the mac address is known.

I figured it was dodgy. How would you init the MAC address earlier though? I've observed the xen devices being initialised at slightly different times with respect to other devices which suggests to me that they (and possibly all devices) are being initialised in separate threads. It would mean setting the MAC address before device initialisation yes?

Not sure how correct this is either, but another solution might be to set the MAC address to say 00:00:00:00:00:01 when the netdev structure is first created. This lets netconsole initialise since the MAC address isn't all zeros or all 0xFF. By the time any net console logs are transmitted, setup_netfront will have already been called and the correct MAC address set so no packets will actually be sent with the above MAC address. I have attached a patch that does this.

Cheers,

Brad

arch/x86/boot/compressed/piggy.S +1 -1

diff -urpNX linux-2.6.32.7/Documentation/dontdiff linux-2.6.32.7/arch/x86/boot/compressed/piggy.S linux-2.6.32.7-xen-netconsole/arch/x86/boot/compressed/piggy.S
@@ -1,6 +1,6 @@
 .section ".rodata.compressed","a",@progbits
 .globl z_input_len
-z_input_len = 2406261
+z_input_len = 2406344
 .globl z_output_len
 z_output_len = 11827328
 .globl z_extract_offset

arch/x86/boot/zoffset.h +1 -1

diff -urpNX linux-2.6.32.7/Documentation/dontdiff linux-2.6.32.7/arch/x86/boot/zoffset.h linux-2.6.32.7-xen-netconsole/arch/x86/boot/zoffset.h
@@ -3,5 +3,5 @@
 #define ZO_startup_32 0x0000000000000000
 #define ZO_z_extract_offset 0x0000000000905000
 #define ZO_z_extract_offset_negative 0xffffffffff6fb000
-#define ZO_z_input_len 0x000000000024b775
+#define ZO_z_input_len 0x000000000024b7c8
 #define ZO_z_output_len 0x0000000000b47880

drivers/net/xen-netfront.c +15 -0

diff -urpNX linux-2.6.32.7/Documentation/dontdiff linux-2.6.32.7/drivers/net/xen-netfront.c linux-2.6.32.7-xen-netconsole/drivers/net/xen-netfront.c
@@ -989,6 +989,15 @@ err:
 	return work_done;
 }
 
+#ifdef CONFIG_NET_POLL_CONTROLLER
+static void xennet_netpoll(struct net_device *dev)
+{
+	struct netfront_info *np = netdev_priv(dev);
+
+	napi_schedule(&np->napi);
+}
+#endif
+
 static int xennet_change_mtu(struct net_device *dev, int mtu)
 {
 	int max = xennet_can_sg(dev) ? 65535 - ETH_HLEN : ETH_DATA_LEN;

@@ -1113,6 +1122,9 @@ static const struct net_device_ops xenne
 	.ndo_change_mtu	     = xennet_change_mtu,
 	.ndo_set_mac_address = eth_mac_addr,
 	.ndo_validate_addr   = eth_validate_addr,
+#ifdef CONFIG_NET_POLL_CONTROLLER
+	.ndo_poll_controller = xennet_netpoll,
+#endif
 };
 
 static struct net_device * __devinit xennet_create_dev(struct xenbus_device *dev)

@@ -1181,6 +1193,9 @@ static struct net_device * __devinit xen
 
 	np->netdev = netdev;
 
+	/* Set MAC to temporary (non-zero) value so that netconsole initialises */
+	netdev->dev_addr[ETH_ALEN - 1] = 1;
+
 	netif_carrier_off(netdev);
 
 	return netdev;

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Jeremy Fitzhardinge

2010-02-16 01:22 UTC #8 lore raw

Quoted 8 lines · Brad Plant

Actually, I tried something like this first because I first looked at pcnet32.c and e100.c:
disable_irq(dev->irq);
xennet_interrupt(dev->irq, dev);
enable_irq(dev->irq);
The same WARN_ONCE() was triggered.

Given that netconsole works with e1000 (I presume; haven't tested it lately), I guess it needs a close look to see what differs from the netfront case.

Quoted 11 lines

Quoted 7 lines
Quoted 2 lines
3. Initially netconsole would not initialise because the interface didn't yet have a mac address as setup_netfront hadn't yet been called. I changed the module_init() in netconsole.c to late_initcall() if !CONFIG_MODULES&& CONFIG_XEN. Is this allowed and sane?
Probably not. In practice every kernel will have modules and Xen enabled, and it will change netconsole for all users. I guess we probably need to find some way to init the MAC address earlier, or trigger netconsole init once the mac address is known.
I figured it was dodgy. How would you init the MAC address earlier though? I've observed the xen devices being initialised at slightly different times with respect to other devices which suggests to me that they (and possibly all devices) are being initialised in separate threads. It would mean setting the MAC address before device initialisation yes?
Not sure how correct this is either, but another solution might be to set the MAC address to say 00:00:00:00:00:01 when the netdev structure is first created. This lets netconsole initialise since the MAC address isn't all zeros or all 0xFF. By the time any net console logs are transmitted, setup_netfront will have already been called and the correct MAC address set so no packets will actually be sent with the above MAC address. I have attached a patch that does this.

That seems a bit sketchy.

netfront is like a lot of network devices; it needs to query some external source to find its own mac address. Mostly this means querying some eeprom over i2c or something, but in our case it means talking to xenstore to find the configured mac address. I think the problem here is that this happens asynchronously, rather than synchronously in the net device's init routine. One fix might be to synchronously wait for the mac address to be set before returning from the init function.