netpoll: netconsole
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 2 lines · Matt Mackall
This patch uses the netpoll API to transmit kernel printks over UDP for uses similar to serial console.
One common problem I saw with the original netconsole patches was that the low level drivers' poll function was grabbing the driver spinlock, but the driver would otherwere do printk with the spinlock hold (->easy deadlock)
Does your patchkit handle that?
I worked around it in some drivers by not grabbing the lock in the direct access poll path, but it's a bit ugly.
-Andi
P.S.: Also what would be really nice for netconsole would be "kernel ifconfig" similar to what the nfsroot code does. With that it would be actually possible to use it as a full console replacement.
Quoted 11 lines · Andi Kleen
Quoted 2 lines · Matt Mackall
This patch uses the netpoll API to transmit kernel printks over UDP for uses similar to serial console.
One common problem I saw with the original netconsole patches was that the low level drivers' poll function was grabbing the driver spinlock, but the driver would otherwere do printk with the spinlock hold (->easy deadlock)
Does your patchkit handle that?
No, haven't encountered it. Which lock are we talking about, specifically?
Quoted 4 lines
P.S.: Also what would be really nice for netconsole would be "kernel ifconfig" similar to what the nfsroot code does. With that it would be actually possible to use it as a full console replacement.
It ups the interface if necessary and has enough info to build a complete raw packet so if I understand you correctly, it's already there. I start getting netconsole messages immediately after driver/net initcalls.
Signature · 1 line
Matt Mackall : http://www.selenic.com : of or relating to the moon
Quoted 1 line
No, haven't encountered it. Which lock are we talking about, specifically?
The hardware register lock of the low level device drivers.
It's a different lock for each driver.
They take it usually in all functions that access hardware registers, including poll.
Quoted 4 lines
It ups the interface if necessary and has enough info to build a complete raw packet so if I understand you correctly, it's already there. I start getting netconsole messages immediately after driver/net initcalls.
Ok, great.
-Andi
Quoted 5 lines · Andi Kleen
Quoted 1 line
No, haven't encountered it. Which lock are we talking about, specifically?
The hardware register lock of the low level device drivers.
It's a different lock for each driver.
I was afraid you were getting at something like that. I've been developing this on UP and the drivers I've been playing with aren't too chatty so nothing's shown up.
And no, I don't think there's much that can be done to fix it, short of putting in deadlock avoidance of some sort or auditing drivers. We run a risk of infinite recursion if we manage to avoid the deadlock anyway.
My current plan is to hope that such printks are rare except in network driver development, in which case using netconsole or kgdb-o-e is asking for trouble anyway.
Signature · 1 line
Matt Mackall : http://www.selenic.com : of or relating to the moon
Quoted 4 lines · Matt Mackall
And no, I don't think there's much that can be done to fix it, short of putting in deadlock avoidance of some sort or auditing drivers. We run a risk of infinite recursion if we manage to avoid the deadlock anyway.
No, we won't infinately recurse - read the code around "down_trylock(&console_sem)" in printk(). Basically if printk() gets called recursively the data will just be written to the buffer and the console driver won't get called until later.
The netconsole problem is only if the net driver calls printk() with its spinlock held (but when not called from netconsole). Then printk() won't know that it's unsafe to re-enter the network driver.
-Mitch
Quoted 3 lines · Mitchell Blank Jr
The netconsole problem is only if the net driver calls
printk()with its spinlock held (but when not called from netconsole). Thenprintk()won't know that it's unsafe to re-enter the network driver.
BTW, this isn't neccesarily a netconsole-only thing. For instance, has anyone ever audited all of the serial and lp drivers to make sure that nothing they call can call printk() while holding a lock? This sounds fairly serious - we could have any number of simple error cases that would cause a deadlock with the right "console=" setting.
It'd be interesting if we could do something like:
1. For every function that appears as a "struct console -> write()" call, follow every possible code path and make a list of every lock that they can try to acquire exclusively.
2. Then scan the entire code base see if we ever call can printk() while holding that same lock.
I'm cc:'ing the Stanford Checker team... maybe they'd be interested in adding something like this to their automated tests.
-Mitch
Quoted 10 lines · Mitchell Blank Jr
Quoted 3 lines · Mitchell Blank Jr
The netconsole problem is only if the net driver calls
printk()with its spinlock held (but when not called from netconsole). Thenprintk()won't know that it's unsafe to re-enter the network driver.BTW, this isn't neccesarily a netconsole-only thing. For instance, has anyone ever audited all of the serial and lp drivers to make sure that nothing they call can call
printk()while holding a lock? This sounds fairly serious - we could have any number of simple error cases that would cause a deadlock with the right "console=" setting.
I have a spinlock debugging patch somewhere that builds a list of locks held by each process. It could easily be extended to do detecting of recursive locking.
Signature · 1 line
Matt Mackall : http://www.selenic.com : of or relating to the moon
Quoted 12 lines · Mitchell Blank Jr
Quoted 3 lines · Mitchell Blank Jr
The netconsole problem is only if the net driver calls
printk()with its spinlock held (but when not called from netconsole). Thenprintk()won't know that it's unsafe to re-enter the network driver.BTW, this isn't neccesarily a netconsole-only thing. For instance, has anyone ever audited all of the serial and lp drivers to make sure that nothing they call can call
printk()while holding a lock? This sounds fairly serious - we could have any number of simple error cases that would cause a deadlock with the right "console=" setting.
Quoted 6 lines
It'd be interesting if we could do something like:
1. For every function that appears as a "struct console ->write()" call, follow every possible code path and make a list of every lock that they can try to acquire exclusively.
2. Then scan the entire code base see if we ever call canprintk()while holding that same lock.
If a driver calls printk() while in ->write(), printk will fail to acquire the console_sem. The printk output will be buffered in log_buf[]. When ->write() releases console_sem it will then display the buffered text.
So the deadlocks to which you refer can only happen when the console driver calls printk while holding driver locks, in a code path where console_sem is not held.