Read-Only THP causes stalls (commit 10359213d)

BUG Christoffer Dall 5 participants 11 messages May 24, 2015 – May 26, 2015 Memory management

Share & tools

Christoffer Dall

author

2015-05-24 19:33 UTC #0 lore raw

Hi all,

I noticed a regression on my arm64 APM X-Gene system a couple of weeks back. I would occassionally see the system lock up and see RCU stalls during the caching phase of kernbench. I then wrote a small script that does nothing but cache the files
(http://paste.ubuntu.com/11324767/) and ran that in a loop. On a known bad commit (v4.1-rc2), out of 25 boots, I never saw it get past 21 iterations of the loop. I have since tried to run a bisect from v3.19 to v4.0 using 100 iterations as my criteria for a good commit.

This resulted in the following first bad commit:

10359213d05acf804558bda7cc9b8422a828d1cd
(mm: incorporate read-only pages into transparent huge pages, 2015-02-11)

Indeed, running the workload on v4.1-rc4 still produced the behavior, but reverting the above commit gets me through 100 iterations of the loop.

I have not tried to reproduce on an x86 system. Turning on a bunch of kernel debugging features *seems* to hide the problem. My config for the XGene system is defconfig + CONFIG_BRIDGE and CONFIG_POWER_RESET_XGENE.

Please let me know if I can help test patches or other things I can do to help. I'm afraid that by simply reading the patch I didn't see anything obviously wrong with it which would cause this behavior.

Thanks,
-Christoffer

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Kirill A. Shutemov

2015-05-25 10:05 UTC #1 lore raw

Quoted 20 lines · Christoffer Dall

Hi all,
I noticed a regression on my arm64 APM X-Gene system a couple of weeks back. I would occassionally see the system lock up and see RCU stalls during the caching phase of kernbench. I then wrote a small script that does nothing but cache the files
(http://paste.ubuntu.com/11324767/) and ran that in a loop. On a known bad commit (v4.1-rc2), out of 25 boots, I never saw it get past 21 iterations of the loop. I have since tried to run a bisect from v3.19 to v4.0 using 100 iterations as my criteria for a good commit.
This resulted in the following first bad commit:
10359213d05acf804558bda7cc9b8422a828d1cd
(mm: incorporate read-only pages into transparent huge pages, 2015-02-11)
Indeed, running the workload on v4.1-rc4 still produced the behavior, but reverting the above commit gets me through 100 iterations of the loop.

Quoted 8 lines

I have not tried to reproduce on an x86 system. Turning on a bunch of kernel debugging features *seems* to hide the problem. My config for the XGene system is defconfig + CONFIG_BRIDGE and CONFIG_POWER_RESET_XGENE.
Please let me know if I can help test patches or other things I can do to help. I'm afraid that by simply reading the patch I didn't see anything obviously wrong with it which would cause this behavior.

I don't see the problem on x86.

Some backtraces could help to track it down.

Signature · 1 line

 Kirill A. Shutemov

Christoffer Dall

author

2015-05-25 10:19 UTC #2 lore raw

Quoted 26 lines · Kirill A. Shutemov

Quoted 25 lines · Christoffer Dall
Hi all,
I noticed a regression on my arm64 APM X-Gene system a couple of weeks back. I would occassionally see the system lock up and see RCU stalls during the caching phase of kernbench. I then wrote a small script that does nothing but cache the files
(http://paste.ubuntu.com/11324767/) and ran that in a loop. On a known bad commit (v4.1-rc2), out of 25 boots, I never saw it get past 21 iterations of the loop. I have since tried to run a bisect from v3.19 to v4.0 using 100 iterations as my criteria for a good commit.
This resulted in the following first bad commit:
10359213d05acf804558bda7cc9b8422a828d1cd
(mm: incorporate read-only pages into transparent huge pages, 2015-02-11)
Indeed, running the workload on v4.1-rc4 still produced the behavior, but reverting the above commit gets me through 100 iterations of the loop.
I have not tried to reproduce on an x86 system. Turning on a bunch of kernel debugging features *seems* to hide the problem. My config for the XGene system is defconfig + CONFIG_BRIDGE and CONFIG_POWER_RESET_XGENE.

Quoted 5 lines

Quoted 3 lines
Please let me know if I can help test patches or other things I can do to help. I'm afraid that by simply reading the patch I didn't see anything obviously wrong with it which would cause this behavior.
I don't see the problem on x86.

I'm wondering if we could have some weird combination of how the specific architecture works along with these patches...

Quoted 3 lines

Some backtraces could help to track it down.

I don't really get backtraces as the sytem just locks up. But here are some of the RCU stalls as I've observed them on the console:

http://paste.ubuntu.com/11014701/
http://paste.ubuntu.com/11023143/
http://paste.ubuntu.com/11023261/

Occasionally, I also get this error from the SATA system at the moment when I power off the device, but not sure if it is related:

ata1: exception Emask 0x10 SAct 0x0 SErr 0x180000 action 0xe frozen
ata1: irq_stat 0x00400000, PHY RDY changed
ata1: SError: { 10B8B Dispar }

Thanks,
-Christoffer

Andrea Arcangeli

2015-05-25 14:15 UTC #3 lore raw

In reply to Christoffer Dall

Hello Christoffer,

Quoted 20 lines · Christoffer Dall

Hi all,
I noticed a regression on my arm64 APM X-Gene system a couple of weeks back. I would occassionally see the system lock up and see RCU stalls during the caching phase of kernbench. I then wrote a small script that does nothing but cache the files
(http://paste.ubuntu.com/11324767/) and ran that in a loop. On a known bad commit (v4.1-rc2), out of 25 boots, I never saw it get past 21 iterations of the loop. I have since tried to run a bisect from v3.19 to v4.0 using 100 iterations as my criteria for a good commit.
This resulted in the following first bad commit:
10359213d05acf804558bda7cc9b8422a828d1cd
(mm: incorporate read-only pages into transparent huge pages, 2015-02-11)
Indeed, running the workload on v4.1-rc4 still produced the behavior, but reverting the above commit gets me through 100 iterations of the loop.

Quoted 8 lines

I have not tried to reproduce on an x86 system. Turning on a bunch of kernel debugging features *seems* to hide the problem. My config for the XGene system is defconfig + CONFIG_BRIDGE and CONFIG_POWER_RESET_XGENE.
Please let me know if I can help test patches or other things I can do to help. I'm afraid that by simply reading the patch I didn't see anything obviously wrong with it which would cause this behavior.

As further confirmation, could you try:

echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan

and verify the problem goes away without having to revert the patch?

Accordingly you should reproduce much eaiser this way (setting $largevalue to 8192 or something, it doesn't matter).

echo $largevalue > /sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/alloc_sleep_millisecs echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs

Then push the system into swap with some memhog -r1000 xG.

The patch just allows readonly anon pages to be collapsed along with read-write ones, the vma permissions allows it, so they have to be swapcache pages, this is why swap shall be required.

Perhaps there's some arch detail that needs fixing but it'll be easier to track it down once you have a way to reproduce fast.

Thanks!
Andrea

Christoffer Dall

author

2015-05-26 08:08 UTC #4 lore raw

Hi Andrea,

Quoted 28 lines · Andrea Arcangeli

Hello Christoffer,
Quoted 25 lines · Christoffer Dall
Hi all,
I noticed a regression on my arm64 APM X-Gene system a couple of weeks back. I would occassionally see the system lock up and see RCU stalls during the caching phase of kernbench. I then wrote a small script that does nothing but cache the files
(http://paste.ubuntu.com/11324767/) and ran that in a loop. On a known bad commit (v4.1-rc2), out of 25 boots, I never saw it get past 21 iterations of the loop. I have since tried to run a bisect from v3.19 to v4.0 using 100 iterations as my criteria for a good commit.
This resulted in the following first bad commit:
10359213d05acf804558bda7cc9b8422a828d1cd
(mm: incorporate read-only pages into transparent huge pages, 2015-02-11)
Indeed, running the workload on v4.1-rc4 still produced the behavior, but reverting the above commit gets me through 100 iterations of the loop.
I have not tried to reproduce on an x86 system. Turning on a bunch of kernel debugging features *seems* to hide the problem. My config for the XGene system is defconfig + CONFIG_BRIDGE and CONFIG_POWER_RESET_XGENE.

Quoted 7 lines

Quoted 3 lines
Please let me know if I can help test patches or other things I can do to help. I'm afraid that by simply reading the patch I didn't see anything obviously wrong with it which would cause this behavior.
As further confirmation, could you try:
echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan

this returns -EINVAL.

But I'm trying now with:

echo never > /sys/kernel/mm/transparent_hugepage/enabled

Quoted 2 lines

and verify the problem goes away without having to revert the patch?

will let you know, so far so good...

Quoted 9 lines

Accordingly you should reproduce much eaiser this way (setting $largevalue to 8192 or something, it doesn't matter).
echo $largevalue > /sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/alloc_sleep_millisecs echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs
Then push the system into swap with some memhog -r1000 xG.

what is memhog? I couldn't find the utility in Google...

I did try with the above settings and just push a bunch of data into ramfs and tmpfs and indeed the sytem died very quickly (on v4.0-rc4).

Quoted 8 lines

The patch just allows readonly anon pages to be collapsed along with read-write ones, the vma permissions allows it, so they have to be swapcache pages, this is why swap shall be required.
Perhaps there's some arch detail that needs fixing but it'll be easier to track it down once you have a way to reproduce fast.

Yes, would be great to be able to reproduce quickly.

Thanks,
-Christoffer

Marc Zyngier

2015-05-26 10:57 UTC #5 lore raw

On 26/05/15 09:08, Christoffer Dall wrote:

[...]

Quoted 3 lines

Quoted 1 line
Then push the system into swap with some memhog -r1000 xG.
what is memhog? I couldn't find the utility in Google...

This looks to be part of the numactl suite, though Debian doesn't seem to include it in its numactl package...

Thanks,

M.

Signature · 1 line

Jazz is not dead. It just smells funny...

Steve Capper

2015-05-26 14:24 UTC #6 lore raw

In reply to Christoffer Dall

Quoted 70 lines · Christoffer Dall

Hi Andrea,
Quoted 28 lines · Andrea Arcangeli
Hello Christoffer,
Quoted 25 lines · Christoffer Dall
Hi all,
I noticed a regression on my arm64 APM X-Gene system a couple of weeks back. I would occassionally see the system lock up and see RCU stalls during the caching phase of kernbench. I then wrote a small script that does nothing but cache the files
(http://paste.ubuntu.com/11324767/) and ran that in a loop. On a known bad commit (v4.1-rc2), out of 25 boots, I never saw it get past 21 iterations of the loop. I have since tried to run a bisect from v3.19 to v4.0 using 100 iterations as my criteria for a good commit.
This resulted in the following first bad commit:
10359213d05acf804558bda7cc9b8422a828d1cd
(mm: incorporate read-only pages into transparent huge pages, 2015-02-11)
Indeed, running the workload on v4.1-rc4 still produced the behavior, but reverting the above commit gets me through 100 iterations of the loop.
I have not tried to reproduce on an x86 system. Turning on a bunch of kernel debugging features *seems* to hide the problem. My config for the XGene system is defconfig + CONFIG_BRIDGE and CONFIG_POWER_RESET_XGENE.
Quoted 7 lines
Quoted 3 lines
Please let me know if I can help test patches or other things I can do to help. I'm afraid that by simply reading the patch I didn't see anything obviously wrong with it which would cause this behavior.
As further confirmation, could you try:
echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan
this returns -EINVAL.
But I'm trying now with:
echo never > /sys/kernel/mm/transparent_hugepage/enabled
Quoted 2 lines
and verify the problem goes away without having to revert the patch?
will let you know, so far so good...
Quoted 9 lines
Accordingly you should reproduce much eaiser this way (setting $largevalue to 8192 or something, it doesn't matter).
echo $largevalue > /sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/alloc_sleep_millisecs echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs
Then push the system into swap with some memhog -r1000 xG.
what is memhog? I couldn't find the utility in Google...
I did try with the above settings and just push a bunch of data into ramfs and tmpfs and indeed the sytem died very quickly (on v4.0-rc4).
Quoted 5 lines
The patch just allows readonly anon pages to be collapsed along with read-write ones, the vma permissions allows it, so they have to be swapcache pages, this is why swap shall be required.

Quoted 8 lines

Quoted 3 lines
Perhaps there's some arch detail that needs fixing but it'll be easier to track it down once you have a way to reproduce fast.
Yes, would be great to be able to reproduce quickly.
Thanks,
-Christoffer

Hi Christoffer,
I'm trying to reproduce this on hardware here; but have been unable to thus far with 4.1-rc2 on a Xgene and Seattle systems.
Also, I tried the memhog + pages_to_scan suggestion from Andrea.

Maybe a silly question, where is your root filesystem located? Is there anything network mounted?

Cheers,
--
Steve

Christoffer Dall

author

2015-05-26 14:35 UTC #7 lore raw

Hi Steve,

Quoted 65 lines · Steve Capper

Quoted 65 lines
Quoted 26 lines · Christoffer Dall
Quoted 25 lines
Hi all,
I noticed a regression on my arm64 APM X-Gene system a couple of weeks back. I would occassionally see the system lock up and see RCU stalls during the caching phase of kernbench. I then wrote a small script that does nothing but cache the files
(http://paste.ubuntu.com/11324767/) and ran that in a loop. On a known bad commit (v4.1-rc2), out of 25 boots, I never saw it get past 21 iterations of the loop. I have since tried to run a bisect from v3.19 to v4.0 using 100 iterations as my criteria for a good commit.
This resulted in the following first bad commit:
10359213d05acf804558bda7cc9b8422a828d1cd
(mm: incorporate read-only pages into transparent huge pages, 2015-02-11)
Indeed, running the workload on v4.1-rc4 still produced the behavior, but reverting the above commit gets me through 100 iterations of the loop.
I have not tried to reproduce on an x86 system. Turning on a bunch of kernel debugging features *seems* to hide the problem. My config for the XGene system is defconfig + CONFIG_BRIDGE and CONFIG_POWER_RESET_XGENE.
Quoted 7 lines
Quoted 3 lines
Please let me know if I can help test patches or other things I can do to help. I'm afraid that by simply reading the patch I didn't see anything obviously wrong with it which would cause this behavior.
As further confirmation, could you try:
echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan
this returns -EINVAL.
But I'm trying now with:
echo never > /sys/kernel/mm/transparent_hugepage/enabled
Quoted 2 lines
and verify the problem goes away without having to revert the patch?
will let you know, so far so good...
Quoted 9 lines
Accordingly you should reproduce much eaiser this way (setting $largevalue to 8192 or something, it doesn't matter).
echo $largevalue > /sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/alloc_sleep_millisecs echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs
Then push the system into swap with some memhog -r1000 xG.
what is memhog? I couldn't find the utility in Google...
I did try with the above settings and just push a bunch of data into ramfs and tmpfs and indeed the sytem died very quickly (on v4.0-rc4).
Quoted 5 lines
The patch just allows readonly anon pages to be collapsed along with read-write ones, the vma permissions allows it, so they have to be swapcache pages, this is why swap shall be required.

Quoted 8 lines

Quoted 5 lines
Quoted 3 lines
Perhaps there's some arch detail that needs fixing but it'll be easier to track it down once you have a way to reproduce fast.
Yes, would be great to be able to reproduce quickly.
I'm trying to reproduce this on hardware here; but have been unable to thus far with 4.1-rc2 on a Xgene and Seattle systems.

Really? That's concerning. I think Andre mentioned he could reproduce...

How many iterations have you run the caching loop for?

Are you using defconfig? I noticed that turning on debugging features was hiding the problem.

Quoted 1 line

Also, I tried the memhog + pages_to_scan suggestion from Andrea.

Any chance you could send me the memhog tool?

Quoted 4 lines

Maybe a silly question, where is your root filesystem located? Is there anything network mounted?

It's a regular ext4 on the local SATA disk. Ubuntu Trusty.

Thanks,
-Christoffer

Steve Capper

2015-05-26 14:48 UTC #8 lore raw

Quoted 90 lines · Christoffer Dall

Hi Steve,
Quoted 65 lines · Steve Capper
Quoted 65 lines
Quoted 26 lines · Christoffer Dall
Quoted 25 lines
Hi all,
I noticed a regression on my arm64 APM X-Gene system a couple of weeks back. I would occassionally see the system lock up and see RCU stalls during the caching phase of kernbench. I then wrote a small script that does nothing but cache the files
(http://paste.ubuntu.com/11324767/) and ran that in a loop. On a known bad commit (v4.1-rc2), out of 25 boots, I never saw it get past 21 iterations of the loop. I have since tried to run a bisect from v3.19 to v4.0 using 100 iterations as my criteria for a good commit.
This resulted in the following first bad commit:
10359213d05acf804558bda7cc9b8422a828d1cd
(mm: incorporate read-only pages into transparent huge pages, 2015-02-11)
Indeed, running the workload on v4.1-rc4 still produced the behavior, but reverting the above commit gets me through 100 iterations of the loop.
I have not tried to reproduce on an x86 system. Turning on a bunch of kernel debugging features *seems* to hide the problem. My config for the XGene system is defconfig + CONFIG_BRIDGE and CONFIG_POWER_RESET_XGENE.
Quoted 7 lines
Quoted 3 lines
Please let me know if I can help test patches or other things I can do to help. I'm afraid that by simply reading the patch I didn't see anything obviously wrong with it which would cause this behavior.
As further confirmation, could you try:
echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan
this returns -EINVAL.
But I'm trying now with:
echo never > /sys/kernel/mm/transparent_hugepage/enabled
Quoted 2 lines
and verify the problem goes away without having to revert the patch?
will let you know, so far so good...
Quoted 9 lines
Accordingly you should reproduce much eaiser this way (setting $largevalue to 8192 or something, it doesn't matter).
echo $largevalue > /sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/alloc_sleep_millisecs echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs
Then push the system into swap with some memhog -r1000 xG.
what is memhog? I couldn't find the utility in Google...
I did try with the above settings and just push a bunch of data into ramfs and tmpfs and indeed the sytem died very quickly (on v4.0-rc4).
Quoted 5 lines
The patch just allows readonly anon pages to be collapsed along with read-write ones, the vma permissions allows it, so they have to be swapcache pages, this is why swap shall be required.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Perhaps there's some arch detail that needs fixing but it'll be easier to track it down once you have a way to reproduce fast.
Yes, would be great to be able to reproduce quickly.
I'm trying to reproduce this on hardware here; but have been unable to thus far with 4.1-rc2 on a Xgene and Seattle systems.
Really? That's concerning. I think Andre mentioned he could reproduce...
How many iterations have you run the caching loop for?
Are you using defconfig? I noticed that turning on debugging features was hiding the problem.
Quoted 1 line
Also, I tried the memhog + pages_to_scan suggestion from Andrea.
Any chance you could send me the memhog tool?
Quoted 1 line

Quoted 7 lines

Quoted 3 lines
Maybe a silly question, where is your root filesystem located? Is there anything network mounted?
It's a regular ext4 on the local SATA disk. Ubuntu Trusty.
Thanks,
-Christoffer

Sending an email to lakml appears to have been enough to make it hang on the Xgene :-).
The system is completely frozen, not even the serial port works.

On Seattle, I've hit 100 iterations multiple times without any problems.

Investigating...

Cheers,
--
Steve

Andrea Arcangeli

2015-05-26 14:58 UTC #9 lore raw

In reply to Christoffer Dall

Quoted 1 line · Christoffer Dall

Any chance you could send me the memhog tool?

memhog is just the first that come to mind because I got it preinstalled everywhere (I only miss it on cyanogenmod as there's no numactl there... yet).

Anything else would do as well, as long as you allocate lots of anonymous memory (malloc(); bzero() or just write 1 byte every 4k). The tmpfs trick was fine as well as you'd end up swapping the anonymous memory allocated by the running apps.

This would be the python version which I actually used sometime if I couldn't find something preinstalled and I didn't want to install packages.

echo 1 >/proc/sys/vm/overcommit_memory
python
a = "a"
while True:

	a += a

This is the more polished way, I just happen to have it installed everywhere (except the cellphone) so I tend to use it, I think it's simpler to install the numactl package.

https://github.com/numactl/numactl/blob/master/memhog.c

Andrea Arcangeli

2015-05-26 14:42 UTC #10 lore raw

In reply to Christoffer Dall

Quoted 4 lines · Christoffer Dall

Quoted 1 line
echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan
this returns -EINVAL.

Oops sorry, I haven't re-read the code, pages_to_scan 0 does not make sense, it would only be useful for debugging purposes because it doesn't shut off khugepaged entirely, so it is ok that it returns -EINVAL, just it won't allow this debug tweak...

Quoted 8 lines

But I'm trying now with:
echo never > /sys/kernel/mm/transparent_hugepage/enabled
Quoted 2 lines
and verify the problem goes away without having to revert the patch?
will let you know, so far so good...

I only intended to disable khugepaged, to validate the theory it was that patch that made the difference.

Increasing the sleep time is equivalent to set pages_to_scan to 0, so you can use this instead:

echo 3600000 >/sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs echo 3600000 >/sys/kernel/mm/transparent_hugepage/khugepaged/alloc_sleep_millisecs

In addition to knowing if it still happens with THP disables, it's interesting to know also if it happens with THP enabled but khugepaged disabled.

Quoted 1 line

what is memhog? I couldn't find the utility in Google...

Somebody answered, yes it's from numactl.

Quoted 2 lines

I did try with the above settings and just push a bunch of data into ramfs and tmpfs and indeed the sytem died very quickly (on v4.0-rc4).

That's fine, memhog just was a way to hit swap. tmpfs pages aren't candidate for khugepaged THP collapsing, so it'd be perhaps quicker to reproduce with something like memhog that uses anonymous memory but it still happens, as long as you hit swap it's ok.

If other arm don't exhibit this problem, perhaps it has to do with some difference in THP, I recall there were two models for arm.

Thanks,
Andrea