Skip to contentSkip to conversation

is printk() safe within a timekeeper_seq write section?

DISCUSSION Jiri Bohac 5 participants 14 messages Mar 6, 2014 – Mar 28, 2014

lore
Share & tools
lore

Jiri Bohac

author
#0 lore raw

Hi,

I'm looking at the printk call in __timekeeping_inject_sleeptime(), introduced in cb5de2f8
(time: Catch invalid timespec sleep values in __timekeeping_inject_sleeptime)

Is it safe to call printk() while timekeeper_seq is held for writing?

What about this call chain?
printk

    vprintk_emit
      console_unlock
        up(&console_sem)
          __up
	    wake_up_process
	      try_to_wake_up
	        ttwu_do_activate
		  ttwu_activate
		    activate_task
		      enqueue_task
		        enqueue_task_fair
			  hrtick_update
			    hrtick_start_fair
			      hrtick_start_fair
			        get_time
				  ktime_get
				    --> endless loop on
				    read_seqcount_retry(&timekeeper_seq, ...)

It looks like an unlikely but possible deadlock.
Or did I overlook something?

Thanks!

Signature · 2 lines
Jiri Bohac <[email protected]>
SUSE Labs, SUSE CZ
#1 lore raw
Quoted 31 lines · Jiri Bohac

Hi,

I'm looking at the printk call in __timekeeping_inject_sleeptime(), introduced in cb5de2f8
(time: Catch invalid timespec sleep values in __timekeeping_inject_sleeptime)

Is it safe to call printk() while timekeeper_seq is held for writing?

What about this call chain?
printk

    vprintk_emit
      console_unlock
        up(&console_sem)
          __up
	    wake_up_process
	      try_to_wake_up
	        ttwu_do_activate
		  ttwu_activate
		    activate_task
		      enqueue_task
		        enqueue_task_fair
			  hrtick_update
			    hrtick_start_fair
			      hrtick_start_fair
			        get_time
				  ktime_get
				    --> endless loop on
				    read_seqcount_retry(&timekeeper_seq, ...)
Quoted 2 lines

It looks like an unlikely but possible deadlock.
Or did I overlook something?

So I don't think I've seen anything like the above in my testing, but it may just be very hard to get that path to trigger.

I was also surprised the seqlock lockdep enablement changes wouldn't catch this, but Jiri pointed out printk calls lockdep_off in vprintk_emit() - which makes sense as you don't want lockdep splats calling printk and recursing - but is frustrating to have that hole in the checking.

There's a few spots where we do printks with the timekeeping seqlock held, and they're annoyingly nested far enough to make deferring the printk awkward. So I'm half thinking we could have some sort of buffer something like time_printk() could fill and then flush it after the lock is dropped. Then we just need something to warn if any new printks' are added to timekeeping seqlock sequences.

The whole thing makes my head spin a bit, since even if we remove the explicit printks, I'm not sure where else printk might be triggered (like via lockdep warnings, for instance), where it might be unsafe.

Peter/Thomas: Any thoughts on the deferred printk buffer? Does printk already have something like this? Any other ideas here?

thanks
-john

#2 lore raw
Quoted 32 lines · John Stultz
Quoted 31 lines · Jiri Bohac

Hi,

I'm looking at the printk call in __timekeeping_inject_sleeptime(), introduced in cb5de2f8
(time: Catch invalid timespec sleep values in __timekeeping_inject_sleeptime)

Is it safe to call printk() while timekeeper_seq is held for writing?

What about this call chain?
printk

    vprintk_emit
      console_unlock
        up(&console_sem)
          __up
	    wake_up_process
	      try_to_wake_up
	        ttwu_do_activate
		  ttwu_activate
		    activate_task
		      enqueue_task
		        enqueue_task_fair
			  hrtick_update
			    hrtick_start_fair
			      hrtick_start_fair
			        get_time
				  ktime_get
				    --> endless loop on
				    read_seqcount_retry(&timekeeper_seq, ...)
Quoted 5 lines
Quoted 2 lines

It looks like an unlikely but possible deadlock.
Or did I overlook something?

So I don't think I've seen anything like the above in my testing, but it may just be very hard to get that path to trigger.

It's hard, but possible:

CPU0 CPU1

T1 down(&console_sem);
			T2 down(&console_sem);
			   --> preemption or interrupt
			        write_seqcount_begin(&timekeeper_seq);
T1 up(&console_sem);
				down(&console_sem);
				....
				up(&console_sem);
				   wakeup(T2);
				     ....
				     hrtick_update();
Quoted 13 lines

I was also surprised the seqlock lockdep enablement changes wouldn't catch this, but Jiri pointed out printk calls lockdep_off in vprintk_emit() - which makes sense as you don't want lockdep splats calling printk and recursing - but is frustrating to have that hole in the checking.

There's a few spots where we do printks with the timekeeping seqlock held, and they're annoyingly nested far enough to make deferring the printk awkward. So I'm half thinking we could have some sort of buffer something like time_printk() could fill and then flush it after the lock is dropped. Then we just need something to warn if any new printks' are added to timekeeping seqlock sequences.

Quoted 6 lines

The whole thing makes my head spin a bit, since even if we remove the explicit printks, I'm not sure where else printk might be triggered (like via lockdep warnings, for instance), where it might be unsafe.

Peter/Thomas: Any thoughts on the deferred printk buffer? Does printk already have something like this? Any other ideas here?

I was thinking about something like that for RT as on RT printk is a complete nightmare. It's simple to implement that, but as we know from the RT experience it can lead to painful loss of debug output.

Assume you printk inside such a region, which just fills the dmesg buffer and schedules the delayed output. Now in that same region you run into a deadlock which causes the whole machine to freeze. Then you won't see the debug output, which might actually give you the hint why the system deadlocked ....

Thanks,

tglx

#3 lore raw
Quoted 76 lines · Thomas Gleixner
Quoted 32 lines · John Stultz
Quoted 31 lines · Jiri Bohac

Hi,

I'm looking at the printk call in __timekeeping_inject_sleeptime(), introduced in cb5de2f8
(time: Catch invalid timespec sleep values in __timekeeping_inject_sleeptime)

Is it safe to call printk() while timekeeper_seq is held for writing?

What about this call chain?
printk

    vprintk_emit
      console_unlock
        up(&console_sem)
          __up
	    wake_up_process
	      try_to_wake_up
	        ttwu_do_activate
		  ttwu_activate
		    activate_task
		      enqueue_task
		        enqueue_task_fair
			  hrtick_update
			    hrtick_start_fair
			      hrtick_start_fair
			        get_time
				  ktime_get
				    --> endless loop on
				    read_seqcount_retry(&timekeeper_seq, ...)
Quoted 4 lines
Quoted 2 lines

It looks like an unlikely but possible deadlock.
Or did I overlook something?

So I don't think I've seen anything like the above in my testing, but it may just be very hard to get that path to trigger.

It's hard, but possible:

CPU0 CPU1

T1 down(&console_sem);
			T2 down(&console_sem);
			   --> preemption or interrupt
			        write_seqcount_begin(&timekeeper_seq);
T1 up(&console_sem);
				down(&console_sem);
				....
				up(&console_sem);
				   wakeup(T2);
				     ....
				     hrtick_update();
Quoted 13 lines

I was also surprised the seqlock lockdep enablement changes wouldn't catch this, but Jiri pointed out printk calls lockdep_off in vprintk_emit() - which makes sense as you don't want lockdep splats calling printk and recursing - but is frustrating to have that hole in the checking.

There's a few spots where we do printks with the timekeeping seqlock held, and they're annoyingly nested far enough to make deferring the printk awkward. So I'm half thinking we could have some sort of buffer something like time_printk() could fill and then flush it after the lock is dropped. Then we just need something to warn if any new printks' are added to timekeeping seqlock sequences.

Quoted 6 lines

The whole thing makes my head spin a bit, since even if we remove the explicit printks, I'm not sure where else printk might be triggered (like via lockdep warnings, for instance), where it might be unsafe.

Peter/Thomas: Any thoughts on the deferred printk buffer? Does printk already have something like this? Any other ideas here?

I was thinking about something like that for RT as on RT printk is a complete nightmare. It's simple to implement that, but as we know from the RT experience it can lead to painful loss of debug output.

Quoted 5 lines

Assume you printk inside such a region, which just fills the dmesg buffer and schedules the delayed output. Now in that same region you run into a deadlock which causes the whole machine to freeze. Then you won't see the debug output, which might actually give you the hint why the system deadlocked ....

Ok, so a generic solution is probably not going to be worth it then. My thought was that since we do a very limited amount of informational printks in the timekeeping code, we can be fairly safe delaying the print-out until we drop the locks.

For timekeeping, its really 4 call sites:
* invalid inject_sleep_time deltas
* > 11% clocksource freq adjustments
* insert leap second
* delete leap second

The first can probably be dropped all together, and signaled via an error return.

The second can probably be replaced by a hard cap, although it would be good to still know we hit that limit.

The last two (only one of which ever really happens) are mostly just informative to the system admin, as most users don't check the TIME_OOP flag to note the leap occurred.

So unless you have other suggestions I think delaying the printouts it is the best approach for the timekeeping code.

We then just have to hope we don't have any other call-out sites that conditionally trip printks as well (nor any that might add printks any time in the future). Any clue on how best to catch those cases? Audits like Jiri's here probably won't scale over time.

thanks
-john

#4 lore raw
Quoted 11 lines · John Stultz

Ok, so a generic solution is probably not going to be worth it then. My thought was that since we do a very limited amount of informational printks in the timekeeping code, we can be fairly safe delaying the print-out until we drop the locks.

For timekeeping, its really 4 call sites:
* invalid inject_sleep_time deltas
* > 11% clocksource freq adjustments
* insert leap second
* delete leap second

There is the printk_sched() you can reuse; you might want to change the name and the generic [sched_delayed] prefix it gets though.

author
#5 lore raw
In reply to John Stultz
Quoted 10 lines · John Stultz

Ok, so a generic solution is probably not going to be worth it then. My thought was that since we do a very limited amount of informational printks in the timekeeping code, we can be fairly safe delaying the print-out until we drop the locks.

For timekeeping, its really 4 call sites:
* invalid inject_sleep_time deltas
* > 11% clocksource freq adjustments
* insert leap second
* delete leap second

I believe these last two were made safe by commit ca4523cd (timekeeping: Shorten seq_count region).

write_seqcount_begin(&timekeeper_seq) is now done after the accumulate_nsecs_to_secs(tk) from where the printks are called.

Regards,

Signature · 2 lines
Jiri Bohac <[email protected]>
SUSE Labs, SUSE CZ
#6 lore raw
Quoted 11 lines · Jiri Bohac
Quoted 10 lines · John Stultz

Ok, so a generic solution is probably not going to be worth it then. My thought was that since we do a very limited amount of informational printks in the timekeeping code, we can be fairly safe delaying the print-out until we drop the locks.

For timekeeping, its really 4 call sites:
* invalid inject_sleep_time deltas
* > 11% clocksource freq adjustments
* insert leap second
* delete leap second

Quoted 5 lines

I believe these last two were made safe by commit ca4523cd (timekeeping: Shorten seq_count region).

write_seqcount_begin(&timekeeper_seq) is now done after the accumulate_nsecs_to_secs(tk) from where the printks are called.

So I started looking into deferring the printk with a small local buffer, but I suddenly realized there are all these WARN_ON's around which would likely have the same problem, no?

So I'm starting to doubt we can safely get away with a timekeeping specific hack to defer the printk. :(

thanks
-john

#7 lore raw
In reply to Thomas Gleixner
Quoted 7 lines · Thomas Gleixner
Quoted 2 lines

Peter/Thomas: Any thoughts on the deferred printk buffer? Does printk already have something like this? Any other ideas here?

I was thinking about something like that for RT as on RT printk is a complete nightmare. It's simple to implement that, but as we know from the RT experience it can lead to painful loss of debug output.

Quoted 5 lines

Assume you printk inside such a region, which just fills the dmesg buffer and schedules the delayed output. Now in that same region you run into a deadlock which causes the whole machine to freeze. Then you won't see the debug output, which might actually give you the hint why the system deadlocked ....

Ok so I started writing a rant that I don't give a crap about klogd and that deferring that wakeup would be perfectly fine; then I looked at the code and found that we in fact do this already.

wake_up_klogd() schedules a lazy irqwork to go wake up, so that's out.

That leaves the console sem wakeup; but I suppose we could redo this patch:

lkml.kernel.org/r/[email protected]

to get rid of that one.

However, at that point we run into the fact that many console drivers do wakeups themselves. I did fix 8250, because that is in fact the only console I really care about, but in general Linus said to give up and deal with the fact that console drivers suck already (or something along those lines).

And while I was looking at all that; I got reminded that I really need to respin this one:

lkml.kernel.org/r/[email protected]

Since that whole printk recursion + zap_locks thing is terminally broken.

#8 lore raw
Quoted 18 lines · Peter Zijlstra
Quoted 7 lines · Thomas Gleixner
Quoted 2 lines

Peter/Thomas: Any thoughts on the deferred printk buffer? Does printk already have something like this? Any other ideas here?

I was thinking about something like that for RT as on RT printk is a complete nightmare. It's simple to implement that, but as we know from the RT experience it can lead to painful loss of debug output.

Quoted 5 lines

Assume you printk inside such a region, which just fills the dmesg buffer and schedules the delayed output. Now in that same region you run into a deadlock which causes the whole machine to freeze. Then you won't see the debug output, which might actually give you the hint why the system deadlocked ....

Ok so I started writing a rant that I don't give a crap about klogd and that deferring that wakeup would be perfectly fine; then I looked at the code and found that we in fact do this already.

Quoted 8 lines

wake_up_klogd() schedules a lazy irqwork to go wake up, so that's out.

That leaves the console sem wakeup; but I suppose we could redo this patch:

lkml.kernel.org/r/[email protected]

to get rid of that one.

I don't know if you've noticed but there's also the following patch:
https://lkml.org/lkml/2013/12/23/310
which would make it pretty easy to just add messages to printk buffer in
timer code and schedule printing later using irq work.

Regarding your referenced patch - the way it is written, it would make
all printk users spin on console_sem->lock all the time while we are flushing buffer to console. I don't think we want that - we trylock the console_sem exactly so that other printk users can proceed while one poor guy is pushing stuff to console.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#9 lore raw
Quoted 24 lines · Jan Kara
Quoted 23 lines · Peter Zijlstra
Quoted 7 lines · Thomas Gleixner
Quoted 2 lines

Peter/Thomas: Any thoughts on the deferred printk buffer? Does printk already have something like this? Any other ideas here?

I was thinking about something like that for RT as on RT printk is a complete nightmare. It's simple to implement that, but as we know from the RT experience it can lead to painful loss of debug output.

Quoted 5 lines

Assume you printk inside such a region, which just fills the dmesg buffer and schedules the delayed output. Now in that same region you run into a deadlock which causes the whole machine to freeze. Then you won't see the debug output, which might actually give you the hint why the system deadlocked ....

Ok so I started writing a rant that I don't give a crap about klogd and that deferring that wakeup would be perfectly fine; then I looked at the code and found that we in fact do this already.

wake_up_klogd() schedules a lazy irqwork to go wake up, so that's out.

That leaves the console sem wakeup; but I suppose we could redo this patch:

Quoted 7 lines
Quoted 3 lines

lkml.kernel.org/r/[email protected]

to get rid of that one.

I don't know if you've noticed but there's also the following patch:
https://lkml.org/lkml/2013/12/23/310
which would make it pretty easy to just add messages to printk buffer in
timer code and schedule printing later using irq work.

Yeah; I suppose that one is prettier.

Quoted 5 lines

Regarding your referenced patch - the way it is written, it would make
all printk users spin on console_sem->lock all the time while we are flushing buffer to console. I don't think we want that - we trylock the console_sem exactly so that other printk users can proceed while one poor guy is pushing stuff to console.

That should be fixable though; just keep enough state for the other printk()s to see they don't need to also flush.

But the idea is to not do the sleep+wakeup dance.

But as stated; that's not going to actually matter much, since the popular console drivers are crap and do wakeups too.

#10 lore raw
Quoted 27 lines · Peter Zijlstra
Quoted 26 lines · Jan Kara
Quoted 25 lines · Peter Zijlstra
Quoted 7 lines · Thomas Gleixner
Quoted 2 lines

Peter/Thomas: Any thoughts on the deferred printk buffer? Does printk already have something like this? Any other ideas here?

I was thinking about something like that for RT as on RT printk is a complete nightmare. It's simple to implement that, but as we know from the RT experience it can lead to painful loss of debug output.

Quoted 5 lines

Assume you printk inside such a region, which just fills the dmesg buffer and schedules the delayed output. Now in that same region you run into a deadlock which causes the whole machine to freeze. Then you won't see the debug output, which might actually give you the hint why the system deadlocked ....

Ok so I started writing a rant that I don't give a crap about klogd and that deferring that wakeup would be perfectly fine; then I looked at the code and found that we in fact do this already.

wake_up_klogd() schedules a lazy irqwork to go wake up, so that's out.

That leaves the console sem wakeup; but I suppose we could redo this patch:

lkml.kernel.org/r/[email protected]

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

to get rid of that one.

I don't know if you've noticed but there's also the following patch:
https://lkml.org/lkml/2013/12/23/310
which would make it pretty easy to just add messages to printk buffer in
timer code and schedule printing later using irq work.

Yeah; I suppose that one is prettier.

JFYI, that patch has been just added to -mm tree.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#11 lore raw
In reply to Thomas Gleixner
Quoted 25 lines · Thomas Gleixner
Quoted 13 lines · John Stultz

I was also surprised the seqlock lockdep enablement changes wouldn't catch this, but Jiri pointed out printk calls lockdep_off in vprintk_emit() - which makes sense as you don't want lockdep splats calling printk and recursing - but is frustrating to have that hole in the checking.

There's a few spots where we do printks with the timekeeping seqlock held, and they're annoyingly nested far enough to make deferring the printk awkward. So I'm half thinking we could have some sort of buffer something like time_printk() could fill and then flush it after the lock is dropped. Then we just need something to warn if any new printks' are added to timekeeping seqlock sequences.

Quoted 6 lines

The whole thing makes my head spin a bit, since even if we remove the explicit printks, I'm not sure where else printk might be triggered (like via lockdep warnings, for instance), where it might be unsafe.

Peter/Thomas: Any thoughts on the deferred printk buffer? Does printk already have something like this? Any other ideas here?

I was thinking about something like that for RT as on RT printk is a complete nightmare. It's simple to implement that, but as we know from the RT experience it can lead to painful loss of debug output.

Quoted 5 lines

Assume you printk inside such a region, which just fills the dmesg buffer and schedules the delayed output. Now in that same region you run into a deadlock which causes the whole machine to freeze. Then you won't see the debug output, which might actually give you the hint why the system deadlocked ....

Certainly just adding messages to printk buffer is more prone to loosing
the messages on error. OTOH if we schedule something like irq work to do the printing (as currently happens with scheduler messages), loosing messages should be relatively rare, shouldn't it?

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#12 lore raw
In reply to John Stultz
Quoted 2 lines · John Stultz

So I don't think I've seen anything like the above in my testing, but it may just be very hard to get that path to trigger.

You have to explicitly enable HRTICK for this to be possible; its disabled by default.

#13 lore raw
In reply to Jiri Bohac
Quoted 31 lines · Jiri Bohac

Hi,

I'm looking at the printk call in __timekeeping_inject_sleeptime(), introduced in cb5de2f8
(time: Catch invalid timespec sleep values in __timekeeping_inject_sleeptime)

Is it safe to call printk() while timekeeper_seq is held for writing?

What about this call chain?
printk

    vprintk_emit
      console_unlock
        up(&console_sem)
          __up
	    wake_up_process
	      try_to_wake_up
	        ttwu_do_activate
		  ttwu_activate
		    activate_task
		      enqueue_task
		        enqueue_task_fair
			  hrtick_update
			    hrtick_start_fair
			      hrtick_start_fair
			        get_time
				  ktime_get
				    --> endless loop on
				    read_seqcount_retry(&timekeeper_seq, ...)
Quoted 2 lines

It looks like an unlikely but possible deadlock.
Or did I overlook something?

So HRTICK is disabled by default; but yes this is a problem when you enable it.

The problem is that our timers are in an entirely different clock base than our (scheduler) clocks.

On SNB+ hardware with TSC deadline timers we have the same clockbase I suppose but I'm not sure that's something we can (and want) to rely on.

Now, I've long had the idea to mess up the hrtimer code to optimize this particular timer, but I'm not entirely sure we can get around having to use the timerkeeper_seq thing, we simply have to get into timer clock space :/