console stalled, cause Hard LOCKUP.

Hello,

Let's switch to email.

Cc-ing Petr and Steven

Thanks, will take a look.

In the meantime, could you check this message:
https://marc.info/?l=linux-kernel&m=152016852803716

It lists several commit IDs that you might backport and try out. Thanks in advance!

	-ss

On (03/20/18 09:34), [email protected] wrote:
[..]

Hm, okay... But "touch_nmi_watchdog() everywhere printk/console-related" is not exactly where I wanted us to be.

By the way e480af09c49736848f749a43dff2c902104f6691 is from 2006. Are you sure you meant exactly that commit? What kernel do you use?

Are you saying that none of Steven's patches helped on your setups?

Hm, printk_deferred is a bit dangerous; it moves console_unlock() to IRQ. So you still can have the problem of stuck CPUs, it's just now you shut up the watchdog. Did you test Steven's patches?

A tricky part about printk_deferred() is that it does not use hand off mechanism. And even more... What we have with "printk vs printk" sceanrio

CPU0 CPU1 ... CPUN

printk printk
console_unlock hand off printk
console_unlock hand off
console_unlock

turns into a good old "one CPU prints it all" when we have "printk vs printk_deferred" case. Because printk_deferred just log_store messages and then _may be_ it grabs the console_sem from IRQ and invokes console_unlock().

So it's something like this

	CPU0			CPU1		...		CPUN

	printk			printk_deffered
	 console_unlock						printk_deferred
	 console_unlock
	 console_unlock
	...			...				...
				printk_deffered			printk_deferred
	 console_unlock
	 console_unlock

// offtopic "I can has printk_kthread?"

You now touch_nmi_watchdog() from the console driver [well... at least this is what e480af09c4973 is doing, but I'm not sure I see how come you didn't have it applied], so that's why you don't see hard lockups on that CPU0. But your printing CPU still can stuck, which will defer RCUs on that CPU, etc. etc. etc. So I'd say that those two approaches

printk_deferred + touch_nmi_watchdog

combined can do quite some harm. One thing for sure - they don't really fix any problems.

	-ss

Cc-ing the kernel list and printk people.

Wen Yang, any chance we can switch to email? Bugzilla is not very handful.

So you are testing printk_kthread now, right? May I ask why? Did Steven's patch help?

	-ss

I'll Cc blockdev

You are right.

Looking at backtraces (https://bugzilla.kernel.org/attachment.cgi?id=274953&action=edit)
there *probably* was just one CPU doing all printk-s and all printouts. And there was one CPU waiting for that printing CPU to unlock the queue spin_lock.

The printing CPU was looping in scsi_request_fn() picking up requests and calling sdev_printk() for each of them, because the device was offline. Given that serial console is not very fast, that we called serial console under queue spin_lock and the number of printks called, it was enough to lockup the CPU which was spining on queue spin_lock and to hard lockup the system.

scsi_request_fn() does unlock the queue lock later, but not in that !scsi_device_online(sdev) error case.

scsi_request_fn()

{
	for (;;) {
		int rtn;
		/*
		 * get next queueable request.  We do this early to make sure
		 * that the request is fully prepared even if we cannot
		 * accept it.
		 */
		req = blk_peek_request(q);
		if (!req)
			break;

		if (unlikely(!scsi_device_online(sdev))) {
			sdev_printk(KERN_ERR, sdev,
				    "rejecting I/O to offline device\n");
			scsi_kill_request(req, q);
			continue;
			^^^^^^^^^ still under spinlock
		}
}

I'd probably just unlock/lock queue lock, rather than ratelimit printk-s, before `continue'. Dunno.

James, Martin, what do you think?

	-ss