Skip to contentSkip to conversation

udiskd high CPU usage with 4.0 git

DISCUSSION Prakash Punnoor 3 participants 7 messages Mar 8, 2015 – Mar 16, 2015

lore
Share & tools
lore
#0 lore raw

Hi,

I noticed the udisks daemon (version 2.1.4) suddenly started using high cpu (one core at 100%) with linux 4.0 git kernel. I bisected it to:

750f199ee8b578062341e6ddfe36c59ac8ff2dcb

And reverting it from current master (at 2cf3afcd4cbe0e32b8722fc291e9255de1b4d6c6) fixes my problem indeed.

I attached dmesg and config from 2cf3afcd4cbe0e32b8722fc291e9255de1b4d6c6 with 750f199ee8b578062341e6ddfe36c59ac8ff2dcb reverted. Any more infos needed? I am actually using a raid5 array, if that matters:

Personalities : [raid6] [raid5] [raid4]
md127 : active raid5 sdd1[1] sdb1[3] sdc1[0]

      3907023872 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3]

[UUU]

unused devices: <none>

The array uses systemd automount feature (x-systemd.automount).

Please CC me, as I am not subscribed.

Regards,

Prakash

#1 lore raw
Quoted 23 lines · On Sun, 08 Mar 2015 18:14:39 +0100 Prakash Punnoor <[email protected]> wrote:

Hi,

I noticed the udisks daemon (version 2.1.4) suddenly started using high cpu (one core at 100%) with linux 4.0 git kernel. I bisected it to:

750f199ee8b578062341e6ddfe36c59ac8ff2dcb

And reverting it from current master (at 2cf3afcd4cbe0e32b8722fc291e9255de1b4d6c6) fixes my problem indeed.

I attached dmesg and config from 2cf3afcd4cbe0e32b8722fc291e9255de1b4d6c6 with 750f199ee8b578062341e6ddfe36c59ac8ff2dcb reverted. Any more infos needed? I am actually using a raid5 array, if that matters:

Personalities : [raid6] [raid5] [raid4]
md127 : active raid5 sdd1[1] sdb1[3] sdc1[0]

      3907023872 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3]

[UUU]

unused devices: <none>

Quoted 8 lines

The array uses systemd automount feature (x-systemd.automount).

Please CC me, as I am not subscribed.

Regards,

Prakash

Thanks for the report.
I can't reproduce this, or see what would cause it.

Can you please reproduce the problem and then run

strace -o /tmp/udisks.trace -f -p pidof udisksd

for a few seconds, then interrupt and post the resulting '/tmp/udisks.trace'.

Thanks,
NeilBrown

#2 lore raw
Quoted 34 lines · NeilBrown
Quoted 23 lines · On Sun, 08 Mar 2015 18:14:39 +0100 Prakash Punnoor <[email protected]> wrote:

Hi,

I noticed the udisks daemon (version 2.1.4) suddenly started using high cpu (one core at 100%) with linux 4.0 git kernel. I bisected it to:

750f199ee8b578062341e6ddfe36c59ac8ff2dcb

And reverting it from current master (at 2cf3afcd4cbe0e32b8722fc291e9255de1b4d6c6) fixes my problem indeed.

I attached dmesg and config from 2cf3afcd4cbe0e32b8722fc291e9255de1b4d6c6 with 750f199ee8b578062341e6ddfe36c59ac8ff2dcb reverted. Any more infos needed? I am actually using a raid5 array, if that matters:

Personalities : [raid6] [raid5] [raid4]
md127 : active raid5 sdd1[1] sdb1[3] sdc1[0]

      3907023872 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3]

[UUU]

unused devices: <none>

Quoted 8 lines

The array uses systemd automount feature (x-systemd.automount).

Please CC me, as I am not subscribed.

Regards,

Prakash

Quoted 8 lines

Thanks for the report.
I can't reproduce this, or see what would cause it.

Can you please reproduce the problem and then run

strace -o /tmp/udisks.trace -f -p pidof udisksd

for a few seconds, then interrupt and post the resulting '/tmp/udisks.trace'.

Hi, attached the trace.

Regards,

Prakash

#3 lore raw
Quoted 38 lines · On Mon, 09 Mar 2015 16:46:41 +0100 Prakash Punnoor <[email protected]> wrote:
Quoted 37 lines · NeilBrown
Quoted 23 lines · On Sun, 08 Mar 2015 18:14:39 +0100 Prakash Punnoor <[email protected]> wrote:

Hi,

I noticed the udisks daemon (version 2.1.4) suddenly started using high cpu (one core at 100%) with linux 4.0 git kernel. I bisected it to:

750f199ee8b578062341e6ddfe36c59ac8ff2dcb

And reverting it from current master (at 2cf3afcd4cbe0e32b8722fc291e9255de1b4d6c6) fixes my problem indeed.

I attached dmesg and config from 2cf3afcd4cbe0e32b8722fc291e9255de1b4d6c6 with 750f199ee8b578062341e6ddfe36c59ac8ff2dcb reverted. Any more infos needed? I am actually using a raid5 array, if that matters:

Personalities : [raid6] [raid5] [raid4]
md127 : active raid5 sdd1[1] sdb1[3] sdc1[0]

      3907023872 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3]

[UUU]

unused devices: <none>

Quoted 8 lines

The array uses systemd automount feature (x-systemd.automount).

Please CC me, as I am not subscribed.

Regards,

Prakash

Thanks for the report.
I can't reproduce this, or see what would cause it.

Quoted 9 lines
Quoted 5 lines

Can you please reproduce the problem and then run

strace -o /tmp/udisks.trace -f -p pidof udisksd

for a few seconds, then interrupt and post the resulting '/tmp/udisks.trace'.

Hi, attached the trace.

Thanks.
The significant part is:

329 poll([{fd=4, events=POLLIN}, {fd=8, events=0}, {fd=9, events=0}, {fd=11, events=POLLIN}, {fd=13, events=0}, {fd=12, events=0}], 6, 513377) = 2 ([{fd=4, revents=POLLIN}, {fd=12, revents=POLLERR}])

repeating frequently, so fd4 and fd12 are always ready. I don't know about fd4, but fd12 is:

329 read(12, "idle\n", 1024) = 5

so that is .../md/sync_action
It is simply always readable. I thought I had checked that, but it seems not.

Anyway I have a fix which I will post shortly.

thanks again,
NeilBrown

#4 lore raw
In reply to NeilBrown
Quoted 8 lines · NeilBrown
Quoted 6 lines · On Sun, 08 Mar 2015 18:14:39 +0100 Prakash Punnoor <[email protected]> wrote:

Hi,

I noticed the udisks daemon (version 2.1.4) suddenly started using high cpu (one core at 100%) with linux 4.0 git kernel. I bisected it to:

750f199ee8b578062341e6ddfe36c59ac8ff2dcb

I had the same problem upgrading from 4.0-rc1 to 4.0-rc3. I have just finished bisecting and "fixing" it.

My bisect points to the same commit.

Looking at udisksd with strace sees a loop of polling and then accessing several md related sysfs files.
The only file that udisksd monitors and was changes by that commit was "sync_action".

If I revert this part of the commit, my system works normal again:

static struct md_sysfs_entry md_scan_mode =
- __ATTR_PREALLOC(sync_action, S_IRUGO|S_IWUSR, action_show, action_store);
+ __ATTR(sync_action, S_IRUGO|S_IWUSR, action_show, action_store);

It seems that polling is broken for peralloc files.

The cause seems to be that kernfs_seq_show() updates ->event, while the new sysfs_kf_read() does not.
So the polling will always trigger and udisksd goes into an inifinite loop looking for changes that are not there.

I fixed my local system by copying the line "of->event = atomic_read(&of->kn->attr.open->event);" from kernfs_seq_show() into sysfs_kf_read(). (I also needed to move the definition of struct kernfs_open_node from kernfs/file.c to kefs-internal.h)

udisksd now again behaves normal, but I'm not sending this change as a patch, because I do not know about the locking and livetime of these objects to evaluate, if that is really the correct fix.

Thanks,

Torsten

#5 lore raw
Quoted 33 lines · On Sat, 14 Mar 2015 21:16:51 +0100 Torsten Kaiser…
Quoted 8 lines · NeilBrown
Quoted 6 lines · On Sun, 08 Mar 2015 18:14:39 +0100 Prakash Punnoor <[email protected]> wrote:

Hi,

I noticed the udisks daemon (version 2.1.4) suddenly started using high cpu (one core at 100%) with linux 4.0 git kernel. I bisected it to:

750f199ee8b578062341e6ddfe36c59ac8ff2dcb

I had the same problem upgrading from 4.0-rc1 to 4.0-rc3. I have just finished bisecting and "fixing" it.

My bisect points to the same commit.

Looking at udisksd with strace sees a loop of polling and then accessing several md related sysfs files.
The only file that udisksd monitors and was changes by that commit was "sync_action".

If I revert this part of the commit, my system works normal again:

static struct md_sysfs_entry md_scan_mode =
- __ATTR_PREALLOC(sync_action, S_IRUGO|S_IWUSR, action_show, action_store);
+ __ATTR(sync_action, S_IRUGO|S_IWUSR, action_show, action_store);

It seems that polling is broken for peralloc files.

The cause seems to be that kernfs_seq_show() updates ->event, while the new sysfs_kf_read() does not.
So the polling will always trigger and udisksd goes into an inifinite loop looking for changes that are not there.

Quoted 8 lines

I fixed my local system by copying the line "of->event = atomic_read(&of->kn->attr.open->event);" from kernfs_seq_show() into sysfs_kf_read(). (I also needed to move the definition of struct kernfs_open_node from kernfs/file.c to kefs-internal.h)

udisksd now again behaves normal, but I'm not sending this change as a patch, because I do not know about the locking and livetime of these objects to evaluate, if that is really the correct fix.

Thanks for the bisection and analysis! Always easier when someone else does the hard work :-)

There is a much simpler patch (as you probably suspected). I'll post it in a moment.

Thank,
NeilBrown

#6 lore raw
Quoted 11 lines · NeilBrown
Quoted 3 lines · On Sat, 14 Mar 2015 21:16:51 +0100 Torsten Kaiser…

udisksd now again behaves normal, but I'm not sending this change as a patch, because I do not know about the locking and livetime of these objects to evaluate, if that is really the correct fix.

Thanks for the bisection and analysis! Always easier when someone else does the hard work :-)

There is a much simpler patch (as you probably suspected). I'll post it in a moment.

Linux-4.0-rc4 is still broken as expected, but after applying your patch from "[PATCH] kernfs: handle poll correctly on 'direct_read' files" my udisksd process behaves normal again.

Thanks for the quick answer + fix!

Torsten