Skip to contentSkip to conversation

BUG: lock held when returning to user space

DISCUSSION Frank Munzert 5 participants 6 messages Mar 12, 2008 – Mar 13, 2008

lore
Share & tools
lore
#0 lore raw

Hi Ingo,

we provided a device driver vmur dealing with z/VM virtual unit record devices (reader, punch, printer). A corresponding user space tool provides functions similar to the CMS commands RECEIVE, PUNCH, PRINT. Unit record devices are not meant for concurrent read or write by multiple users, that's why we need to serialize access. The driver's open method uses mutex_trylock or mutex_lock_interruptible to ensure exclusive access to the device, while its release method uses mutex_unlock.

As a consequence, lockdep complains about locks being held when returning to user space. We used a very simple char device driver (appended below) to produce this message: ================================================ [ BUG: lock held when returning to user space! ]
------------------------------------------------
testapp/2683 is leaving the kernel with locks still held!
1 lock held by testapp/2683:
#0: (&test_mutex){--..}, at: [<000003e00003316c>] test_open+0x30/0x64 [test]

For the vmur device driver it is crucial to have only one process access a given unit record device node at a given time. So having open hold the mutex and return to user space is exactly what we want. Is there any annotation to tell lockdep to suppress or bypass this kind of warning?

Thanks in advance,

Frank

#include  <linux/module.h>
#include  <linux/init.h>
#include  <linux/fs.h>
#include  <linux/types.h>
#include  <linux/cdev.h>
#define  DEVICE_NAME  "test"
#define  MODULE_NAME  "test"

static dev_t Devno;
static struct cdev test_cdev;
static struct mutex test_mutex;

static ssize_t test_read(struct file *filp, char *buff, size_t len,
loff_t *whence)

{
	return 1;
}

static ssize_t test_write(struct file *filp, const char *buff,
size_t len, loff_t *whence)

{
	return 1;
}

static int test_open(struct inode *ino, struct file *filp)

{
	if (mutex_lock_interruptible(&test_mutex))
		return -ERESTARTSYS;
	printk("%s: test device is open.\n", MODULE_NAME);
	return 0;
}

static int test_close(struct inode *ino, struct file *filp)

{
	printk("%s: test device is closed.\n", MODULE_NAME);
	mutex_unlock(&test_mutex);
	return 0;
}

static struct file_operations Fops = { .owner = THIS_MODULE,

				         .read    = test_read,
				         .write   = test_write,
				         .open    = test_open,
				         .release = test_close    };

static int __init test_init(void)

{
	int rc;
	rc = alloc_chrdev_region(&Devno, 0, 1, DEVICE_NAME);
	if (rc < 0)
	{
		printk("%s: Registration failed...\n",MODULE_NAME);
		return rc;
	}
	printk("%s: Registration %s at major number %d\n", MODULE_NAME,
				DEVICE_NAME, MAJOR(Devno));
	cdev_init(&test_cdev, &Fops);
	test_cdev.owner = THIS_MODULE;
	rc = cdev_add(&test_cdev, Devno, 1);
	if (rc < 0) {
		printk("%s: Device object not added!\n", MODULE_NAME);
		unregister_chrdev_region(Devno, 1);
		return rc;
	}
	printk("%s: Device added.\n", MODULE_NAME);
	mutex_init(&test_mutex);
	return 0;
}

static void __exit test_exit(void)

{
	cdev_del(&test_cdev);
	unregister_chrdev_region(Devno, 1);
	printk("%s: Module removed.\n", MODULE_NAME);
}

module_init(test_init);
module_exit(test_exit);

MODULE_LICENSE("GPL");
MODULE_AUTHOR("Frank");
MODULE_DESCRIPTION("Simple char driver.");

#1 lore raw
Quoted 8 lines · Frank Munzert

we provided a device driver vmur dealing with z/VM virtual unit record devices (reader, punch, printer). A corresponding user space tool provides functions similar to the CMS commands RECEIVE, PUNCH, PRINT. Unit record devices are not meant for concurrent read or write by multiple users, that's why we need to serialize access. The driver's open method uses mutex_trylock or mutex_lock_interruptible to ensure exclusive access to the device, while its release method uses mutex_unlock.

snip.

Quoted 4 lines

For the vmur device driver it is crucial to have only one process access a given unit record device node at a given time. So having open hold the mutex and return to user space is exactly what we want. Is there any annotation to tell lockdep to suppress or bypass this kind of warning?

This sounds like a serious abuse of mutexes.

Wouldn't it be correct to use the mutex to protect a separate variable (which indicates whether the device has been open()ed) and nothing else? Then there is no need to hold the mutex across the syscalls and open() can simply fail if the separate variable is set.

Kind regards,
Vegard Nossum

#2 lore raw
In reply to Frank Munzert
Quoted 5 lines · Frank Munzert

Unit record devices are not meant for concurrent read or write by multiple users, that's why we need to serialize access. The driver's open method uses mutex_trylock or mutex_lock_interruptible to ensure exclusive access to the device, while its release method uses mutex_unlock.

Which is wrong usage of mutex.

Quoted 5 lines

As a consequence, lockdep complains about locks being held when returning to user space. We used a very simple char device driver (appended below) to produce this message:
------------------------------------------------
testapp/2683 is leaving the kernel with locks still held!

[ ... ]

Quoted 3 lines

For the vmur device driver it is crucial to have only one process access a given unit record device node at a given time. So having open hold the mutex and return to user space is exactly what we want.

No, it is not. You probably want just to have a single-bit flag somewhere, and update/check it atomically, see below.

Quoted 2 lines

Is there any annotation to tell lockdep to suppress or bypass this kind of warning?

No, as it is a real bug if you use mutexes in this way. What happens if process that has called open() on your device (and has not closed it yet) calls fork()?
Another breakage scenario -- what if the filedescriptor is sent through unix socket to another process? etc.

Look at test_and_set_bit_lock() and clear_bit_unlock(), those you want to use.

Signature · 2 lines
Jiri Kosina
SUSE Labs
#3 lore raw
Quoted 5 lines · Jiri Kosina

No, as it is a real bug if you use mutexes in this way. What happens if process that has called open() on your device (and has not closed it yet) calls fork()?
Another breakage scenario -- what if the filedescriptor is sent through unix socket to another process? etc.

There's a number of places where a semaphore is used across system calls.

for instance the usb skeleton,
drivers/usb/usb-skeleton.c

Several of the watchdog drivers,

	drivers/watchdog/s3c2410_wdt.c

These need to be removed, but the usage is clearly not compatible with the mutex API ..

If you convert them to atomic counts then you loose the sleeping aspect of the semaphore, which you'd have to add back somehow.

The only API that seems straight forward is using complete's .. Then you get an atomic count and all the sleeping function calls you might want.. (include/linux/completion.h) The problem with complete's is that you can't start them out at "1" or "completed" unless you actually run complete() once during initialization (that's kind of ugly) ..

Daniel

#4 lore raw
Quoted 20 lines · Daniel Walker
Quoted 5 lines · Jiri Kosina

No, as it is a real bug if you use mutexes in this way. What happens if process that has called open() on your device (and has not closed it yet) calls fork()?
Another breakage scenario -- what if the filedescriptor is sent through unix socket to another process? etc.

There's a number of places where a semaphore is used across system calls.

for instance the usb skeleton,
drivers/usb/usb-skeleton.c

Several of the watchdog drivers,

	drivers/watchdog/s3c2410_wdt.c

These need to be removed, but the usage is clearly not compatible with the mutex API ..

Quoted 8 lines

If you convert them to atomic counts then you loose the sleeping aspect of the semaphore, which you'd have to add back somehow.

The only API that seems straight forward is using complete's .. Then you get an atomic count and all the sleeping function calls you might want.. (include/linux/completion.h) The problem with complete's is that you can't start them out at "1" or "completed" unless you actually run complete() once during initialization (that's kind of ugly) ..

A simple waitqueue would work.

#5 lore raw
Quoted 6 lines · Peter Zijlstra
Quoted 6 lines

These need to be removed, but the usage is clearly not compatible with the mutex API ..

If you convert them to atomic counts then you loose the sleeping aspect of the semaphore, which you'd have to add back somehow.

Quoted 7 lines
Quoted 5 lines

The only API that seems straight forward is using complete's .. Then you get an atomic count and all the sleeping function calls you might want.. (include/linux/completion.h) The problem with complete's is that you can't start them out at "1" or "completed" unless you actually run complete() once during initialization (that's kind of ugly) ..

A simple waitqueue would work.

I was just mulling over what the most straight forward way would be.. I thought about wait queues, but the API didn't seem as easy to use a completes..

Daniel