Skip to contentSkip to conversation

mutex

PATCH Amos Kong 2 participants 5 messages Sep 9, 2014 – Sep 10, 2014

Subsystems (3)
lore
Share & tools
lore

Amos Kong

author
#0 lore raw

Hi Amit, Rusty

RHBZ: https://bugzilla.redhat.com/show_bug.cgi?id=1127062
steps:
- Read random data by 'dd if=/dev/hwrng of=/dev/null' in guest
- check sysfs files in the same time, 'cat /sys/class/misc/hw_random/rng_*'

Result: cat process will get stuck, it will return if we kill dd process.

We have some static variables (eg, current_rng, data_avail, etc) in hw_random/core.c, they are protected by rng_mutex. I try to workaround this issue by undelay(100) after mutex_unlock() in rng_dev_read(). This gives chance for hwrng_attr_*_show() to get mutex.

This patch also contains some cleanup, moving some code out of mutex protection.

Do you have some suggestion? Thanks.

drivers/char/hw_random/core.c +5 -4
index aa30a25..fa69020 100644
@@ -194,6 +194,7 @@ static ssize_t rng_dev_read(struct file *filp, char __user *buf,
 		}
 
 		mutex_unlock(&rng_mutex);
+		udelay(100);
 
 		if (need_resched())
 			schedule_timeout_interruptible(1);
@@ -233,10 +234,10 @@ static ssize_t hwrng_attr_current_store(struct device *dev,
 	int err;
 	struct hwrng *rng;
 
+	err = -ENODEV;
 	err = mutex_lock_interruptible(&rng_mutex);
 	if (err)
 		return -ERESTARTSYS;
-	err = -ENODEV;
 	list_for_each_entry(rng, &rng_list, list) {
 		if (strcmp(rng->name, buf) == 0) {
 			if (rng == current_rng) {
@@ -270,8 +271,8 @@ static ssize_t hwrng_attr_current_show(struct device *dev,
 		return -ERESTARTSYS;
 	if (current_rng)
 		name = current_rng->name;
-	ret = snprintf(buf, PAGE_SIZE, "%s\n", name);
 	mutex_unlock(&rng_mutex);
+	ret = snprintf(buf, PAGE_SIZE, "%s\n", name);
 
 	return ret;
 }
@@ -284,19 +285,19 @@ static ssize_t hwrng_attr_available_show(struct device *dev,
 	ssize_t ret = 0;
 	struct hwrng *rng;
 
+	buf[0] = '\0';
 	err = mutex_lock_interruptible(&rng_mutex);
 	if (err)
 		return -ERESTARTSYS;
-	buf[0] = '\0';
 	list_for_each_entry(rng, &rng_list, list) {
 		strncat(buf, rng->name, PAGE_SIZE - ret - 1);
 		ret += strlen(rng->name);
 		strncat(buf, " ", PAGE_SIZE - ret - 1);
 		ret++;
 	}
+	mutex_unlock(&rng_mutex);
 	strncat(buf, "\n", PAGE_SIZE - ret - 1);
 	ret++;
-	mutex_unlock(&rng_mutex);
 
 	return ret;
 }
author
#1 lore raw

RFC virtio-rng: fail to read sysfs of a busy device

(Resend to fix the subject)

Hi Amit, Rusty

RHBZ: https://bugzilla.redhat.com/show_bug.cgi?id=1127062
steps:
- Read random data by 'dd if=/dev/hwrng of=/dev/null' in guest
- check sysfs files in the same time, 'cat /sys/class/misc/hw_random/rng_*'

Result: cat process will get stuck, it will return if we kill dd process.

We have some static variables (eg, current_rng, data_avail, etc) in hw_random/core.c, they are protected by rng_mutex. I try to workaround this issue by undelay(100) after mutex_unlock() in rng_dev_read(). This gives chance for hwrng_attr_*_show() to get mutex.

This patch also contains some cleanup, moving some code out of mutex protection.

Do you have some suggestion? Thanks.

drivers/char/hw_random/core.c +5 -4
index aa30a25..fa69020 100644
@@ -194,6 +194,7 @@ static ssize_t rng_dev_read(struct file *filp, char __user *buf,
 		}
 
 		mutex_unlock(&rng_mutex);
+		udelay(100);
 
 		if (need_resched())
 			schedule_timeout_interruptible(1);
@@ -233,10 +234,10 @@ static ssize_t hwrng_attr_current_store(struct device *dev,
 	int err;
 	struct hwrng *rng;
 
+	err = -ENODEV;
 	err = mutex_lock_interruptible(&rng_mutex);
 	if (err)
 		return -ERESTARTSYS;
-	err = -ENODEV;
 	list_for_each_entry(rng, &rng_list, list) {
 		if (strcmp(rng->name, buf) == 0) {
 			if (rng == current_rng) {
@@ -270,8 +271,8 @@ static ssize_t hwrng_attr_current_show(struct device *dev,
 		return -ERESTARTSYS;
 	if (current_rng)
 		name = current_rng->name;
-	ret = snprintf(buf, PAGE_SIZE, "%s\n", name);
 	mutex_unlock(&rng_mutex);
+	ret = snprintf(buf, PAGE_SIZE, "%s\n", name);
 
 	return ret;
 }
@@ -284,19 +285,19 @@ static ssize_t hwrng_attr_available_show(struct device *dev,
 	ssize_t ret = 0;
 	struct hwrng *rng;
 
+	buf[0] = '\0';
 	err = mutex_lock_interruptible(&rng_mutex);
 	if (err)
 		return -ERESTARTSYS;
-	buf[0] = '\0';
 	list_for_each_entry(rng, &rng_list, list) {
 		strncat(buf, rng->name, PAGE_SIZE - ret - 1);
 		ret += strlen(rng->name);
 		strncat(buf, " ", PAGE_SIZE - ret - 1);
 		ret++;
 	}
+	mutex_unlock(&rng_mutex);
 	strncat(buf, "\n", PAGE_SIZE - ret - 1);
 	ret++;
-	mutex_unlock(&rng_mutex);
 
 	return ret;
 }
Signature · 1 line
			Amos.
#2 lore raw

Re: RFC virtio-rng: fail to read sysfs of a busy device

Quoted 10 lines · Amos Kong

(Resend to fix the subject)

Hi Amit, Rusty

RHBZ: https://bugzilla.redhat.com/show_bug.cgi?id=1127062
steps:
- Read random data by 'dd if=/dev/hwrng of=/dev/null' in guest
- check sysfs files in the same time, 'cat /sys/class/misc/hw_random/rng_*'

Result: cat process will get stuck, it will return if we kill dd process.

How common is it going to be to have a long-running 'dd' process on /dev/hwrng?

Also, with the new khwrng thread, reading from /dev/hwrng isn't required -- just use /dev/random?

(This doesn't mean we shouldn't fix the issue here...)

Quoted 16 lines

We have some static variables (eg, current_rng, data_avail, etc) in hw_random/core.c, they are protected by rng_mutex. I try to workaround this issue by undelay(100) after mutex_unlock() in rng_dev_read(). This gives chance for hwrng_attr_*_show() to get mutex.

This patch also contains some cleanup, moving some code out of mutex protection.

Do you have some suggestion? Thanks.

drivers/char/hw_random/core.c +0 -0
index aa30a25..fa69020 100644

drivers/char/hw_random/core.c static ssize_t rng_dev_read(struct file *filp, char __user *buf, · line 197

Quoted 4 lines
drivers/char/hw_random/core.c +1 -0
@@ -194,6 +194,7 @@ static ssize_t rng_dev_read(struct file *filp, char __user *buf,
 		}
 
 		mutex_unlock(&rng_mutex);
+		udelay(100);

We have a need_resched() right below. Why doesn't that work?

drivers/char/hw_random/core.c static ssize_t hwrng_attr_current_store(struct device *dev, · line 235

Quoted 5 lines
drivers/char/hw_random/core.c +0 -0
 		if (need_resched())
 			schedule_timeout_interruptible(1);
@@ -233,10 +234,10 @@ static ssize_t hwrng_attr_current_store(struct device *dev,
 	int err;
 	struct hwrng *rng;

The following hunk doesn't work:

drivers/char/hw_random/core.c static ssize_t hwrng_attr_current_store(struct device *dev, · line 238

Quoted 2 lines
drivers/char/hw_random/core.c +1 -0
+	err = -ENODEV;
 	err = mutex_lock_interruptible(&rng_mutex);

err is being set to another value in the next line!

drivers/char/hw_random/core.c static ssize_t hwrng_attr_current_store(struct device *dev, · line 239

Quoted 3 lines
drivers/char/hw_random/core.c +0 -1
 	if (err)
 		return -ERESTARTSYS;
-	err = -ENODEV;

And all usage of err below now won't have -ENODEV but some other value.

drivers/char/hw_random/core.c static ssize_t hwrng_attr_current_show(struct device *dev, · line 275

Quoted 10 lines
drivers/char/hw_random/core.c +1 -1
 	list_for_each_entry(rng, &rng_list, list) {
 		if (strcmp(rng->name, buf) == 0) {
 			if (rng == current_rng) {
@@ -270,8 +271,8 @@ static ssize_t hwrng_attr_current_show(struct device *dev,
 		return -ERESTARTSYS;
 	if (current_rng)
 		name = current_rng->name;
-	ret = snprintf(buf, PAGE_SIZE, "%s\n", name);
 	mutex_unlock(&rng_mutex);
+	ret = snprintf(buf, PAGE_SIZE, "%s\n", name);

This looks OK...

Quoted 13 lines
drivers/char/hw_random/core.c +1 -1
 	return ret;
 }
@@ -284,19 +285,19 @@ static ssize_t hwrng_attr_available_show(struct device *dev,
 	ssize_t ret = 0;
 	struct hwrng *rng;
 
+	buf[0] = '\0';
 	err = mutex_lock_interruptible(&rng_mutex);
 	if (err)
 		return -ERESTARTSYS;
 
-	buf[0] = '\0';

drivers/char/hw_random/core.c static ssize_t hwrng_attr_available_show(struct device *dev, · line 300

Quoted 10 lines
drivers/char/hw_random/core.c +1 -1
 	list_for_each_entry(rng, &rng_list, list) {
 		strncat(buf, rng->name, PAGE_SIZE - ret - 1);
 		ret += strlen(rng->name);
 		strncat(buf, " ", PAGE_SIZE - ret - 1);
 		ret++;
 	}
+	mutex_unlock(&rng_mutex);
 	strncat(buf, "\n", PAGE_SIZE - ret - 1);
 	ret++;
-	mutex_unlock(&rng_mutex);

But this isn't resulting in savings; the majority of the time is being spent in the for loop, and that writes to the buffer.

BTW I don't expect strcat'ing to the buf in each of these scenarios is a long operation, so this reworking doesn't strike to me as something we should pursue.

Amit

author
#3 lore raw

Re: RFC virtio-rng: fail to read sysfs of a busy device

Quoted 10 lines · Amit Shah
Quoted 9 lines · Amos Kong

(Resend to fix the subject)

Hi Amit, Rusty

RHBZ: https://bugzilla.redhat.com/show_bug.cgi?id=1127062
steps:
- Read random data by 'dd if=/dev/hwrng of=/dev/null' in guest
- check sysfs files in the same time, 'cat /sys/class/misc/hw_random/rng_*'

Quoted 4 lines
Quoted 1 line

Result: cat process will get stuck, it will return if we kill dd process.

How common is it going to be to have a long-running 'dd' process on /dev/hwrng?

Not a common usage, but we have this strict testing.

Quoted 2 lines

Also, with the new khwrng thread, reading from /dev/hwrng isn't required -- just use /dev/random?

Yes.

Quoted 1 line

(This doesn't mean we shouldn't fix the issue here...)

Completely agree :-)

Quoted 18 lines
Quoted 18 lines

We have some static variables (eg, current_rng, data_avail, etc) in hw_random/core.c, they are protected by rng_mutex. I try to workaround this issue by undelay(100) after mutex_unlock() in rng_dev_read(). This gives chance for hwrng_attr_*_show() to get mutex.

This patch also contains some cleanup, moving some code out of mutex protection.

Do you have some suggestion? Thanks.

drivers/char/hw_random/core.c +0 -0
index aa30a25..fa69020 100644
@@ -194,6 +194,7 @@ static ssize_t rng_dev_read(struct file *filp, char __user *buf,
 		}
 
Quoted 4 lines

drivers/char/hw_random/core.c static ssize_t rng_dev_read(struct file *filp, char __user *buf, · line 197

Quoted 2 lines
drivers/char/hw_random/core.c +1 -0
 		mutex_unlock(&rng_mutex);
+		udelay(100);

We have a need_resched() right below. Why doesn't that work?

need_resched() is giving chance for userspace to

Quoted 1 line

drivers/char/hw_random/core.c static ssize_t rng_dev_read(struct file *filp, char __user *buf, · line 198

Quoted 1 line
drivers/char/hw_random/core.c +0 -0
 		if (need_resched())

It never success in my debugging.

If we remove this check and always call schedule_timeout_interruptible(1), problem also disappears.

drivers/char/hw_random/core.c +0 -0
index aa30a25..263a370 100644
@@ -195,8 +195,7 @@ static ssize_t rng_dev_read(struct file *filp,

char __user *buf,

                mutex_unlock(&rng_mutex);

- if (need_resched())
- schedule_timeout_interruptible(1);
+ schedule_timeout_interruptible(1);

                if (signal_pending(current)) {
                        err = -ERESTARTSYS;
Quoted 12 lines

drivers/char/hw_random/core.c static ssize_t hwrng_attr_current_store(struct device *dev, · line 235

Quoted 4 lines
drivers/char/hw_random/core.c +0 -0
 			schedule_timeout_interruptible(1);
@@ -233,10 +234,10 @@ static ssize_t hwrng_attr_current_store(struct device *dev,
 	int err;
 	struct hwrng *rng;

The following hunk doesn't work:

drivers/char/hw_random/core.c static ssize_t hwrng_attr_current_store(struct device *dev, · line 237

Quoted 2 lines
drivers/char/hw_random/core.c +1 -0
+	err = -ENODEV;
 	err = mutex_lock_interruptible(&rng_mutex);

err is being set to another value in the next line!

Quoted 5 lines

drivers/char/hw_random/core.c static ssize_t hwrng_attr_current_store(struct device *dev, · line 238

Quoted 3 lines
drivers/char/hw_random/core.c +0 -1
 	if (err)
 		return -ERESTARTSYS;
-	err = -ENODEV;

And all usage of err below now won't have -ENODEV but some other value.

Oops!

Quoted 25 lines

drivers/char/hw_random/core.c static ssize_t hwrng_attr_current_show(struct device *dev, · line 275

Quoted 10 lines
drivers/char/hw_random/core.c +1 -1
 	list_for_each_entry(rng, &rng_list, list) {
 		if (strcmp(rng->name, buf) == 0) {
 			if (rng == current_rng) {
@@ -270,8 +271,8 @@ static ssize_t hwrng_attr_current_show(struct device *dev,
 		return -ERESTARTSYS;
 	if (current_rng)
 		name = current_rng->name;
-	ret = snprintf(buf, PAGE_SIZE, "%s\n", name);
 	mutex_unlock(&rng_mutex);
+	ret = snprintf(buf, PAGE_SIZE, "%s\n", name);

This looks OK...

Quoted 12 lines
drivers/char/hw_random/core.c +1 -0
 	return ret;
 }
@@ -284,19 +285,19 @@ static ssize_t hwrng_attr_available_show(struct device *dev,
 	ssize_t ret = 0;
 	struct hwrng *rng;
 
+	buf[0] = '\0';
 	err = mutex_lock_interruptible(&rng_mutex);
 	if (err)
 		return -ERESTARTSYS;
 
Quoted 14 lines

drivers/char/hw_random/core.c static ssize_t hwrng_attr_available_show(struct device *dev, · line 300

Quoted 11 lines
drivers/char/hw_random/core.c +1 -2
-	buf[0] = '\0';
 	list_for_each_entry(rng, &rng_list, list) {
 		strncat(buf, rng->name, PAGE_SIZE - ret - 1);
 		ret += strlen(rng->name);
 		strncat(buf, " ", PAGE_SIZE - ret - 1);
 		ret++;
 	}
+	mutex_unlock(&rng_mutex);
 	strncat(buf, "\n", PAGE_SIZE - ret - 1);
 	ret++;
-	mutex_unlock(&rng_mutex);

But this isn't resulting in savings; the majority of the time is being spent in the for loop, and that writes to the buffer.

Right

Quoted 5 lines

BTW I don't expect strcat'ing to the buf in each of these scenarios is a long operation, so this reworking doesn't strike to me as something we should pursue.

Amit

Signature · 1 line
			Amos.
author
#4 lore raw

Re: RFC virtio-rng: fail to read sysfs of a busy device

Quoted 11 lines · Amos Kong
Quoted 10 lines · Amit Shah
Quoted 9 lines · Amos Kong

(Resend to fix the subject)

Hi Amit, Rusty

RHBZ: https://bugzilla.redhat.com/show_bug.cgi?id=1127062
steps:
- Read random data by 'dd if=/dev/hwrng of=/dev/null' in guest
- check sysfs files in the same time, 'cat /sys/class/misc/hw_random/rng_*'

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Result: cat process will get stuck, it will return if we kill dd process.

How common is it going to be to have a long-running 'dd' process on /dev/hwrng?

Not a common usage, but we have this strict testing.

For -smp 1:
It's easy to reproduce with slow backend (/dev/random). cat can return most of time with some delay if we use quick backend (/dev/urandom).

But for -smp 2:
I didn't touch this problem even with slow backend.

Quoted 31 lines
Quoted 2 lines

Also, with the new khwrng thread, reading from /dev/hwrng isn't required -- just use /dev/random?

Yes.

Quoted 1 line

(This doesn't mean we shouldn't fix the issue here...)

Completely agree :-)

Quoted 22 lines
Quoted 16 lines

We have some static variables (eg, current_rng, data_avail, etc) in hw_random/core.c, they are protected by rng_mutex. I try to workaround this issue by undelay(100) after mutex_unlock() in rng_dev_read(). This gives chance for hwrng_attr_*_show() to get mutex.

This patch also contains some cleanup, moving some code out of mutex protection.

Do you have some suggestion? Thanks.

drivers/char/hw_random/core.c +0 -0
index aa30a25..fa69020 100644

drivers/char/hw_random/core.c static ssize_t rng_dev_read(struct file *filp, char __user *buf, · line 197

Quoted 4 lines
drivers/char/hw_random/core.c +1 -0
@@ -194,6 +194,7 @@ static ssize_t rng_dev_read(struct file *filp, char __user *buf,
 		}
 
 		mutex_unlock(&rng_mutex);
+		udelay(100);

We have a need_resched() right below. Why doesn't that work?

[smp 1]
Why need_resched() always return zero? what's the original purpose of it ?

Quoted 94 lines
Quoted 2 lines

drivers/char/hw_random/core.c static ssize_t rng_dev_read(struct file *filp, char __user *buf, · line 198

Quoted 2 lines
drivers/char/hw_random/core.c +0 -0
 		if (need_resched())

It never success in my debugging.

If we remove this check and always call schedule_timeout_interruptible(1), problem also disappears.

drivers/char/hw_random/core.c +1 -2
index aa30a25..263a370 100644
@@ -195,8 +195,7 @@ static ssize_t rng_dev_read(struct file *filp,
 char __user *buf,
 
                mutex_unlock(&rng_mutex);
 
-               if (need_resched())
-                       schedule_timeout_interruptible(1);
+               schedule_timeout_interruptible(1);
 
                if (signal_pending(current)) {
                        err = -ERESTARTSYS;
Quoted 12 lines

drivers/char/hw_random/core.c static ssize_t hwrng_attr_current_store(struct device *dev, · line 235

Quoted 4 lines
drivers/char/hw_random/core.c +0 -0
 			schedule_timeout_interruptible(1);
@@ -233,10 +234,10 @@ static ssize_t hwrng_attr_current_store(struct device *dev,
 	int err;
 	struct hwrng *rng;

The following hunk doesn't work:

drivers/char/hw_random/core.c static ssize_t hwrng_attr_current_store(struct device *dev, · line 237

Quoted 2 lines
drivers/char/hw_random/core.c +1 -0
+	err = -ENODEV;
 	err = mutex_lock_interruptible(&rng_mutex);

err is being set to another value in the next line!

Quoted 5 lines

drivers/char/hw_random/core.c static ssize_t hwrng_attr_current_store(struct device *dev, · line 238

Quoted 3 lines
drivers/char/hw_random/core.c +0 -1
 	if (err)
 		return -ERESTARTSYS;
-	err = -ENODEV;

And all usage of err below now won't have -ENODEV but some other value.

Oops!

Quoted 25 lines

drivers/char/hw_random/core.c static ssize_t hwrng_attr_current_show(struct device *dev, · line 275

Quoted 10 lines
drivers/char/hw_random/core.c +1 -1
 	list_for_each_entry(rng, &rng_list, list) {
 		if (strcmp(rng->name, buf) == 0) {
 			if (rng == current_rng) {
@@ -270,8 +271,8 @@ static ssize_t hwrng_attr_current_show(struct device *dev,
 		return -ERESTARTSYS;
 	if (current_rng)
 		name = current_rng->name;
-	ret = snprintf(buf, PAGE_SIZE, "%s\n", name);
 	mutex_unlock(&rng_mutex);
+	ret = snprintf(buf, PAGE_SIZE, "%s\n", name);

This looks OK...

Quoted 12 lines
drivers/char/hw_random/core.c +1 -0
 	return ret;
 }
@@ -284,19 +285,19 @@ static ssize_t hwrng_attr_available_show(struct device *dev,
 	ssize_t ret = 0;
 	struct hwrng *rng;
 
+	buf[0] = '\0';
 	err = mutex_lock_interruptible(&rng_mutex);
 	if (err)
 		return -ERESTARTSYS;
 
Quoted 14 lines

drivers/char/hw_random/core.c static ssize_t hwrng_attr_available_show(struct device *dev, · line 300

Quoted 11 lines
drivers/char/hw_random/core.c +1 -2
-	buf[0] = '\0';
 	list_for_each_entry(rng, &rng_list, list) {
 		strncat(buf, rng->name, PAGE_SIZE - ret - 1);
 		ret += strlen(rng->name);
 		strncat(buf, " ", PAGE_SIZE - ret - 1);
 		ret++;
 	}
+	mutex_unlock(&rng_mutex);
 	strncat(buf, "\n", PAGE_SIZE - ret - 1);
 	ret++;
-	mutex_unlock(&rng_mutex);

But this isn't resulting in savings; the majority of the time is being spent in the for loop, and that writes to the buffer.

Right

Quoted 5 lines

BTW I don't expect strcat'ing to the buf in each of these scenarios is a long operation, so this reworking doesn't strike to me as something we should pursue.

Amit

Signature · 1 line
			Amos.
Signature · 1 line
			Amos.