udiskd high CPU usage with 4.0 git
Hi,
I noticed the udisks daemon (version 2.1.4) suddenly started using high cpu (one core at 100%) with linux 4.0 git kernel. I bisected it to:
750f199ee8b578062341e6ddfe36c59ac8ff2dcb
And reverting it from current master (at 2cf3afcd4cbe0e32b8722fc291e9255de1b4d6c6) fixes my problem indeed.
I attached dmesg and config from 2cf3afcd4cbe0e32b8722fc291e9255de1b4d6c6 with 750f199ee8b578062341e6ddfe36c59ac8ff2dcb reverted. Any more infos needed? I am actually using a raid5 array, if that matters:
Personalities : [raid6] [raid5] [raid4]
md127 : active raid5 sdd1[1] sdb1[3] sdc1[0]
3907023872 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3]
[UUU]
unused devices: <none>
The array uses systemd automount feature (x-systemd.automount).
Please CC me, as I am not subscribed.
Regards,
Prakash
Quoted 23 lines · On Sun, 08 Mar 2015 18:14:39 +0100 Prakash Punnoor <[email protected]> wrote:
Hi,
I noticed the udisks daemon (version 2.1.4) suddenly started using high cpu (one core at 100%) with linux 4.0 git kernel. I bisected it to:
750f199ee8b578062341e6ddfe36c59ac8ff2dcbAnd reverting it from current master (at
2cf3afcd4cbe0e32b8722fc291e9255de1b4d6c6) fixes my problem indeed.I attached dmesg and config from
2cf3afcd4cbe0e32b8722fc291e9255de1b4d6c6with750f199ee8b578062341e6ddfe36c59ac8ff2dcbreverted. Any more infos needed? I am actually using a raid5 array, if that matters:Personalities : [raid6] [raid5] [raid4]
md127 : active raid5 sdd1[1] sdb1[3] sdc1[0]3907023872 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3][UUU]
unused devices: <none>
Quoted 8 lines
The array uses systemd automount feature (x-systemd.automount).
Please CC me, as I am not subscribed.
Regards,
Prakash
Thanks for the report.
I can't reproduce this, or see what would cause it.
Can you please reproduce the problem and then run
strace -o /tmp/udisks.trace -f -p pidof udisksd
for a few seconds, then interrupt and post the resulting '/tmp/udisks.trace'.
Thanks,
NeilBrown
Quoted 34 lines · NeilBrown
Quoted 23 lines · On Sun, 08 Mar 2015 18:14:39 +0100 Prakash Punnoor <[email protected]> wrote:
Hi,
I noticed the udisks daemon (version 2.1.4) suddenly started using high cpu (one core at 100%) with linux 4.0 git kernel. I bisected it to:
750f199ee8b578062341e6ddfe36c59ac8ff2dcbAnd reverting it from current master (at
2cf3afcd4cbe0e32b8722fc291e9255de1b4d6c6) fixes my problem indeed.I attached dmesg and config from
2cf3afcd4cbe0e32b8722fc291e9255de1b4d6c6with750f199ee8b578062341e6ddfe36c59ac8ff2dcbreverted. Any more infos needed? I am actually using a raid5 array, if that matters:Personalities : [raid6] [raid5] [raid4]
md127 : active raid5 sdd1[1] sdb1[3] sdc1[0]3907023872 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3][UUU]
unused devices: <none>
Quoted 8 lines
The array uses systemd automount feature (x-systemd.automount).
Please CC me, as I am not subscribed.
Regards,
Prakash
Quoted 8 lines
Thanks for the report.
I can't reproduce this, or see what would cause it.Can you please reproduce the problem and then run
strace -o
/tmp/udisks.trace-f -ppidof udisksdfor a few seconds, then interrupt and post the resulting '
/tmp/udisks.trace'.
Hi, attached the trace.
Regards,
Prakash
Quoted 8 lines · NeilBrown
Quoted 6 lines · On Sun, 08 Mar 2015 18:14:39 +0100 Prakash Punnoor <[email protected]> wrote:
Hi,
I noticed the udisks daemon (version 2.1.4) suddenly started using high cpu (one core at 100%) with linux 4.0 git kernel. I bisected it to:
750f199ee8b578062341e6ddfe36c59ac8ff2dcb
I had the same problem upgrading from 4.0-rc1 to 4.0-rc3. I have just finished bisecting and "fixing" it.
My bisect points to the same commit.
Looking at udisksd with strace sees a loop of polling and then accessing several md related sysfs files.
The only file that udisksd monitors and was changes by that commit was "sync_action".
If I revert this part of the commit, my system works normal again:
static struct md_sysfs_entry md_scan_mode =
- __ATTR_PREALLOC(sync_action, S_IRUGO|S_IWUSR, action_show, action_store);
+ __ATTR(sync_action, S_IRUGO|S_IWUSR, action_show, action_store);
It seems that polling is broken for peralloc files.
The cause seems to be that kernfs_seq_show() updates ->event, while the new sysfs_kf_read() does not.
So the polling will always trigger and udisksd goes into an inifinite loop looking for changes that are not there.
I fixed my local system by copying the line "of->event = atomic_read(&of->kn->attr.open->event);" from kernfs_seq_show() into sysfs_kf_read(). (I also needed to move the definition of struct kernfs_open_node from kernfs/file.c to kefs-internal.h)
udisksd now again behaves normal, but I'm not sending this change as a patch, because I do not know about the locking and livetime of these objects to evaluate, if that is really the correct fix.
Thanks,
Torsten
Quoted 33 lines · On Sat, 14 Mar 2015 21:16:51 +0100 Torsten Kaiser…
Quoted 8 lines · NeilBrown
Quoted 6 lines · On Sun, 08 Mar 2015 18:14:39 +0100 Prakash Punnoor <[email protected]> wrote:
Hi,
I noticed the udisks daemon (version 2.1.4) suddenly started using high cpu (one core at 100%) with linux 4.0 git kernel. I bisected it to:
750f199ee8b578062341e6ddfe36c59ac8ff2dcbI had the same problem upgrading from 4.0-rc1 to 4.0-rc3. I have just finished bisecting and "fixing" it.
My bisect points to the same commit.
Looking at udisksd with strace sees a loop of polling and then accessing several md related sysfs files.
The only file that udisksd monitors and was changes by that commit was "sync_action".If I revert this part of the commit, my system works normal again:
static struct
md_sysfs_entrymd_scan_mode=
-__ATTR_PREALLOC(sync_action, S_IRUGO|S_IWUSR, action_show, action_store);
+__ATTR(sync_action, S_IRUGO|S_IWUSR, action_show, action_store);It seems that polling is broken for peralloc files.
The cause seems to be that
kernfs_seq_show()updates ->event, while the newsysfs_kf_read()does not.
So the polling will always trigger and udisksd goes into an inifinite loop looking for changes that are not there.
Quoted 8 lines
I fixed my local system by copying the line "
of->event=atomic_read(&of->kn->attr.open->event);" fromkernfs_seq_show()intosysfs_kf_read(). (I also needed to move the definition of structkernfs_open_nodefromkernfs/file.ctokefs-internal.h)udisksd now again behaves normal, but I'm not sending this change as a patch, because I do not know about the locking and livetime of these objects to evaluate, if that is really the correct fix.
Thanks for the bisection and analysis! Always easier when someone else does the hard work :-)
There is a much simpler patch (as you probably suspected). I'll post it in a moment.
Thank,
NeilBrown
Quoted 38 lines · On Mon, 09 Mar 2015 16:46:41 +0100 Prakash Punnoor <[email protected]> wrote:
Quoted 37 lines · NeilBrown
Quoted 23 lines · On Sun, 08 Mar 2015 18:14:39 +0100 Prakash Punnoor <[email protected]> wrote:
Hi,
I noticed the udisks daemon (version 2.1.4) suddenly started using high cpu (one core at 100%) with linux 4.0 git kernel. I bisected it to:
750f199ee8b578062341e6ddfe36c59ac8ff2dcbAnd reverting it from current master (at
2cf3afcd4cbe0e32b8722fc291e9255de1b4d6c6) fixes my problem indeed.I attached dmesg and config from
2cf3afcd4cbe0e32b8722fc291e9255de1b4d6c6with750f199ee8b578062341e6ddfe36c59ac8ff2dcbreverted. Any more infos needed? I am actually using a raid5 array, if that matters:Personalities : [raid6] [raid5] [raid4]
md127 : active raid5 sdd1[1] sdb1[3] sdc1[0]3907023872 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3][UUU]
unused devices: <none>
Quoted 8 lines
The array uses systemd automount feature (x-systemd.automount).
Please CC me, as I am not subscribed.
Regards,
Prakash
Thanks for the report.
I can't reproduce this, or see what would cause it.
Quoted 9 lines
Quoted 5 lines
Can you please reproduce the problem and then run
strace -o
/tmp/udisks.trace-f -ppidof udisksdfor a few seconds, then interrupt and post the resulting '
/tmp/udisks.trace'.Hi, attached the trace.
Thanks.
The significant part is:
329 poll([{fd=4, events=POLLIN}, {fd=8, events=0}, {fd=9, events=0}, {fd=11, events=POLLIN}, {fd=13, events=0}, {fd=12, events=0}], 6, 513377) = 2 ([{fd=4, revents=POLLIN}, {fd=12, revents=POLLERR}])
repeating frequently, so fd4 and fd12 are always ready. I don't know about fd4, but fd12 is:
329 read(12, "idle\n", 1024) = 5
so that is .../md/sync_action
It is simply always readable. I thought I had checked that, but it seems not.
Anyway I have a fix which I will post shortly.
thanks again,
NeilBrown
Quoted 11 lines · NeilBrown
Quoted 3 lines · On Sat, 14 Mar 2015 21:16:51 +0100 Torsten Kaiser…
udisksd now again behaves normal, but I'm not sending this change as a patch, because I do not know about the locking and livetime of these objects to evaluate, if that is really the correct fix.
Thanks for the bisection and analysis! Always easier when someone else does the hard work :-)
There is a much simpler patch (as you probably suspected). I'll post it in a moment.
Linux-4.0-rc4 is still broken as expected, but after applying your patch from "[PATCH] kernfs: handle poll correctly on 'direct_read' files" my udisksd process behaves normal again.
Thanks for the quick answer + fix!
Torsten