BUG: lock held when returning to user space
Hi Ingo,
we provided a device driver vmur dealing with z/VM virtual unit record devices (reader, punch, printer). A corresponding user space tool provides functions similar to the CMS commands RECEIVE, PUNCH, PRINT. Unit record devices are not meant for concurrent read or write by multiple users, that's why we need to serialize access. The driver's open method uses mutex_trylock or mutex_lock_interruptible to ensure exclusive access to the device, while its release method uses mutex_unlock.
As a consequence, lockdep complains about locks being held when returning to user space. We used a very simple char device driver (appended below) to produce this message: ================================================ [ BUG: lock held when returning to user space! ]
------------------------------------------------
testapp/2683 is leaving the kernel with locks still held!
1 lock held by testapp/2683:
#0: (&test_mutex){--..}, at: [<000003e00003316c>] test_open+0x30/0x64 [test]
For the vmur device driver it is crucial to have only one process access a given unit record device node at a given time. So having open hold the mutex and return to user space is exactly what we want. Is there any annotation to tell lockdep to suppress or bypass this kind of warning?
Thanks in advance,
Frank
#include <linux/module.h> #include <linux/init.h> #include <linux/fs.h> #include <linux/types.h> #include <linux/cdev.h>
#define DEVICE_NAME "test" #define MODULE_NAME "test"
static dev_t Devno;
static struct cdev test_cdev;
static struct mutex test_mutex;
static ssize_t test_read(struct file *filp, char *buff, size_t len,
loff_t *whence)
{
return 1;
}static ssize_t test_write(struct file *filp, const char *buff,
size_t len, loff_t *whence)
{
return 1;
}static int test_open(struct inode *ino, struct file *filp)
{
if (mutex_lock_interruptible(&test_mutex))
return -ERESTARTSYS;
printk("%s: test device is open.\n", MODULE_NAME);
return 0;
}static int test_close(struct inode *ino, struct file *filp)
{
printk("%s: test device is closed.\n", MODULE_NAME);
mutex_unlock(&test_mutex);
return 0;
}static struct file_operations Fops = { .owner = THIS_MODULE,
.read = test_read, .write = test_write, .open = test_open, .release = test_close };
static int __init test_init(void)
{
int rc;
rc = alloc_chrdev_region(&Devno, 0, 1, DEVICE_NAME);
if (rc < 0)
{
printk("%s: Registration failed...\n",MODULE_NAME);
return rc;
}
printk("%s: Registration %s at major number %d\n", MODULE_NAME,
DEVICE_NAME, MAJOR(Devno));
cdev_init(&test_cdev, &Fops);
test_cdev.owner = THIS_MODULE;
rc = cdev_add(&test_cdev, Devno, 1);
if (rc < 0) {
printk("%s: Device object not added!\n", MODULE_NAME);
unregister_chrdev_region(Devno, 1);
return rc;
}
printk("%s: Device added.\n", MODULE_NAME);
mutex_init(&test_mutex);
return 0;
}static void __exit test_exit(void)
{
cdev_del(&test_cdev);
unregister_chrdev_region(Devno, 1);
printk("%s: Module removed.\n", MODULE_NAME);
}module_init(test_init);module_exit(test_exit);
MODULE_LICENSE("GPL");MODULE_AUTHOR("Frank");MODULE_DESCRIPTION("Simple char driver.");
Quoted 8 lines · Frank Munzert
we provided a device driver vmur dealing with z/VM virtual unit record devices (reader, punch, printer). A corresponding user space tool provides functions similar to the CMS commands RECEIVE, PUNCH, PRINT. Unit record devices are not meant for concurrent read or write by multiple users, that's why we need to serialize access. The driver's open method uses
mutex_trylockormutex_lock_interruptibleto ensure exclusive access to the device, while its release method usesmutex_unlock.
snip.
Quoted 4 lines
For the vmur device driver it is crucial to have only one process access a given unit record device node at a given time. So having open hold the mutex and return to user space is exactly what we want. Is there any annotation to tell lockdep to suppress or bypass this kind of warning?
This sounds like a serious abuse of mutexes.
Wouldn't it be correct to use the mutex to protect a separate variable (which indicates whether the device has been open()ed) and nothing else? Then there is no need to hold the mutex across the syscalls and open() can simply fail if the separate variable is set.
Kind regards,
Vegard Nossum
Quoted 5 lines · Frank Munzert
Unit record devices are not meant for concurrent read or write by multiple users, that's why we need to serialize access. The driver's open method uses
mutex_trylockormutex_lock_interruptibleto ensure exclusive access to the device, while its release method usesmutex_unlock.
Which is wrong usage of mutex.
Quoted 5 lines
As a consequence, lockdep complains about locks being held when returning to user space. We used a very simple char device driver (appended below) to produce this message:
------------------------------------------------
testapp/2683 is leaving the kernel with locks still held!
[ ... ]
Quoted 3 lines
For the vmur device driver it is crucial to have only one process access a given unit record device node at a given time. So having open hold the mutex and return to user space is exactly what we want.
No, it is not. You probably want just to have a single-bit flag somewhere, and update/check it atomically, see below.
Quoted 2 lines
Is there any annotation to tell lockdep to suppress or bypass this kind of warning?
No, as it is a real bug if you use mutexes in this way. What happens if process that has called open() on your device (and has not closed it yet) calls fork()?
Another breakage scenario -- what if the filedescriptor is sent through unix socket to another process? etc.
Look at test_and_set_bit_lock() and clear_bit_unlock(), those you want to use.
Signature · 2 lines
Jiri Kosina SUSE Labs
Quoted 5 lines · Jiri Kosina
No, as it is a real bug if you use mutexes in this way. What happens if process that has called
open()on your device (and has not closed it yet) callsfork()?
Another breakage scenario -- what if the filedescriptor is sent through unix socket to another process? etc.
There's a number of places where a semaphore is used across system calls.
for instance the usb skeleton,
drivers/usb/usb-skeleton.c
Several of the watchdog drivers,
drivers/watchdog/s3c2410_wdt.c
These need to be removed, but the usage is clearly not compatible with the mutex API ..
If you convert them to atomic counts then you loose the sleeping aspect of the semaphore, which you'd have to add back somehow.
The only API that seems straight forward is using complete's .. Then you get an atomic count and all the sleeping function calls you might want.. (include/linux/completion.h) The problem with complete's is that you can't start them out at "1" or "completed" unless you actually run complete() once during initialization (that's kind of ugly) ..
Daniel
Quoted 20 lines · Daniel Walker
Quoted 5 lines · Jiri Kosina
No, as it is a real bug if you use mutexes in this way. What happens if process that has called
open()on your device (and has not closed it yet) callsfork()?
Another breakage scenario -- what if the filedescriptor is sent through unix socket to another process? etc.There's a number of places where a semaphore is used across system calls.
for instance the usb skeleton,
drivers/usb/usb-skeleton.cSeveral of the watchdog drivers,
drivers/watchdog/s3c2410_wdt.cThese need to be removed, but the usage is clearly not compatible with the mutex API ..
Quoted 8 lines
If you convert them to atomic counts then you loose the sleeping aspect of the semaphore, which you'd have to add back somehow.
The only API that seems straight forward is using complete's .. Then you get an atomic count and all the sleeping function calls you might want.. (
include/linux/completion.h) The problem with complete's is that you can't start them out at "1" or "completed" unless you actually runcomplete()once during initialization (that's kind of ugly) ..
A simple waitqueue would work.
Quoted 6 lines · Peter Zijlstra
Quoted 6 lines
These need to be removed, but the usage is clearly not compatible with the mutex API ..
If you convert them to atomic counts then you loose the sleeping aspect of the semaphore, which you'd have to add back somehow.
Quoted 7 lines
Quoted 5 lines
The only API that seems straight forward is using complete's .. Then you get an atomic count and all the sleeping function calls you might want.. (
include/linux/completion.h) The problem with complete's is that you can't start them out at "1" or "completed" unless you actually runcomplete()once during initialization (that's kind of ugly) ..A simple waitqueue would work.
I was just mulling over what the most straight forward way would be.. I thought about wait queues, but the API didn't seem as easy to use a completes..
Daniel