zswap + zram enabled BUG

Bug 192571 In review Andrew Morton 6 participants 16 messages Jan 17, 2017 – Jan 26, 2017 2 messages not in the archive

Subsystems (2)

Memory management zswap compressed swap caching

lore

Share & tools

lore

Andrew Morton

author maintainer

2017-01-17 20:22 UTC #2 lore raw

In reply to a message not in the archive

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Quoted 67 lines · On Sat, 14 Jan 2017 17:32:04 +0000 [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=192571
--- Comment #1 from Gluzskiy Alexandr <[email protected]> ---
[199961.576604] ------------[ cut here ]------------
[199961.577830] kernel BUG at mm/zswap.c:1108!
[199961.579006] invalid opcode: 0000 [#1] SMP
[199961.580166] Modules linked in: uvcvideo gspca_zc3xx xt_sctp zram ccm
act_mirred ifb sch_ingress cls_u32 sch_sfq sch_htb nf_conntrack_netlink nfnetlink sit tunnel4 ip_tunnel iptable_mangle ipt_REJECT nf_reject_ipv4 xt_recent xt_TCPMSS nf_conntrack_ipv6 nf_defrag_ipv6 iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_conntrack xt_nat xt_tcpudp xt_multiport ip6table_filter ip6table_raw ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_raw ip_tables x_tables radeon ath9k led_class ath9k_common i2c_algo_bit btrfs ath9k_hw ttm mac80211 drm_kms_helper xor snd_hda_codec_via snd_hda_codec_generic cfbfillrect ath syscopyarea cfbimgblt cfg80211 sysfillrect sysimgblt fb_sys_fops cfbcopyarea rfkill drm snd_hda_intel snd_hda_codec r8169 xhci_pci xhci_hcd backlight
[199961.587843]  parport_pc ohci_pci raid6_pq snd_hda_core mii button ohci_hcd
asus_atk0110 i2c_piix4 acpi_cpufreq processor sch_fq_codel br_netfilter bridge stp llc snd_usb_audio snd_hwdep snd_usbmidi_lib snd_pcm snd_rawmidi snd_seq_device snd_timer snd soundcore vhost_net tun nfsd vhost macvtap auth_rpcgss macvlan oid_registry nfs_acl lockd grace kvm_amd kvm irqbypass gspca_main v4l2_common k10temp hwmon videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videodev videobuf2_core i2c_core parport fbcon bitblit softcursor fb fbdev font sunrpc autofs4 [last unloaded: uvcvideo]
[199961.594974] CPU: 2 PID: 2755 Comm: syncthing Not tainted 4.9.2 #4
[199961.596459] Hardware name: System manufacturer System Product Name/M4A77TD,
BIOS 2104    06/28/2010
[199961.597974] task: ffff880035c19680 task.stack: ffffc90000510000
[199961.599490] RIP: 0010:[<ffffffff8112c6c2>]  [<ffffffff8112c6c2>]
zswap_frontswap_load+0x142/0x160
[199961.601042] RSP: 0000:ffffc90000513cb0  EFLAGS: 00010282
[199961.602588] RAX: ffffffff818263a0 RBX: ffff88036b2fb930 RCX:
ffffc90000513c98
[199961.604141] RDX: ffff8801a75da000 RSI: ffff8802ee9d0240 RDI:
ffff88041c25d000
[199961.605687] RBP: ffff880035c19680 R08: ffff8802ee9d0249 R09:
ffff8801a75da0ac
[199961.607240] R10: ffff8801a75db000 R11: ffff8802ee9d027d R12:
00000000ffffffea
[199961.608788] R13: ffff8804176e3830 R14: ffff8804176e3838 R15:
000000c42a6ac008
[199961.610315] FS:  00007fe1fa7fc700(0000) GS:ffff88042fc80000(0000)
knlGS:0000000000000000
[199961.611838] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[199961.613357] CR2: 000000c42a6ac008 CR3: 00000000a8411000 CR4:
00000000000006e0
[199961.614864] Stack:
[199961.616327]  00001000fffffffe ffffffff81823780 000000000014487b
ffffea00069d7680
[199961.617820]  0000000000000001 ffff880418614c00 ffffffff8112b768
ffffea00069d7680
[199961.619310]  ffff880418614c00 000000000014487b 00000000024200ca
ffff88011d834900
[199961.620799] Call Trace:
[199961.622250]  [<ffffffff8112b768>] ? __frontswap_load+0x68/0xc0
[199961.623689]  [<ffffffff8112666c>] ? swap_readpage+0x8c/0x120
[199961.625115]  [<ffffffff81126e61>] ? read_swap_cache_async+0x21/0x40
[199961.626545]  [<ffffffff81126f96>] ? swapin_readahead+0x116/0x1e0
[199961.627973]  [<ffffffff812b704e>] ? radix_tree_lookup_slot+0xe/0x20
[199961.629398]  [<ffffffff8111236f>] ? do_swap_page+0x42f/0x660
[199961.630799]  [<ffffffff81114bca>] ? handle_mm_fault+0x76a/0x1080
[199961.632163]  [<ffffffff811544ec>] ? new_sync_read+0xac/0xe0
[199961.633496]  [<ffffffff8102c7a9>] ? __do_page_fault+0x169/0x3e0
[199961.634798]  [<ffffffff8102ca5b>] ? do_page_fault+0x1b/0x60
[199961.636106]  [<ffffffff810e3cc9>] ?

Quoted 12 lines

__context_tracking_exit.part.1+0x49/0x60
[199961.637424]  [<ffffffff8157c7cf>] ? page_fault+0x1f/0x30
[199961.638739] Code: fb ff ff 41 c6 45 08 00 48 83 c4 08 44 89 e0 5b 5d 41 5c
41 5d 41 5e c3 be 0f 00 00 00 48 c7 c7 12 d6 6d 81 e8 e0 d2 f1 ff eb b0 <0f> 0b
0f 1f 84 00 00 00 00 00 66 2e 0f 1f 84 00 00 00 00 00 66 
[199961.641538] RIP  [<ffffffff8112c6c2>] zswap_frontswap_load+0x142/0x160
[199961.642922]  RSP <ffffc90000513cb0>
[199961.648971] ---[ end trace 76742a0cd4818a78 ]---

Signature · 2 lines

You are receiving this mail because:
You are the assignee for the bug.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Sergey Senozhatsky

2017-01-18 01:39 UTC #3 lore raw

Cc Dan

Quoted 5 lines · Andrew Morton

Quoted 5 lines
https://bugzilla.kernel.org/show_bug.cgi?id=192571
--- Comment #1 from Gluzskiy Alexandr <[email protected]> ---
[199961.576604] ------------[ cut here ]------------
[199961.577830] kernel BUG at mm/zswap.c:1108!

zswap didn't manage to decompress the page:

static int zswap_frontswap_load(unsigned type, pgoff_t offset,
struct page *page)
{
...

	dst = kmap_atomic(page);
	tfm = *get_cpu_ptr(entry->pool->tfm);
	ret = crypto_comp_decompress(tfm, src, entry->length, dst, &dlen);
	put_cpu_ptr(entry->pool->tfm);
	kunmap_atomic(dst);
	zpool_unmap_handle(entry->pool->zpool, entry->handle);
	BUG_ON(ret);
	^^^^^^^^^^^

is there anything suspicious in dmesg?

-ss

[..]

Quoted 40 lines

[199961.596459] Hardware name: System manufacturer System Product Name/M4A77TD,
BIOS 2104    06/28/2010
[199961.597974] task: ffff880035c19680 task.stack: ffffc90000510000
[199961.599490] RIP: 0010:[<ffffffff8112c6c2>]  [<ffffffff8112c6c2>]
zswap_frontswap_load+0x142/0x160
[199961.601042] RSP: 0000:ffffc90000513cb0  EFLAGS: 00010282
[199961.602588] RAX: ffffffff818263a0 RBX: ffff88036b2fb930 RCX:
ffffc90000513c98
[199961.604141] RDX: ffff8801a75da000 RSI: ffff8802ee9d0240 RDI:
ffff88041c25d000
[199961.605687] RBP: ffff880035c19680 R08: ffff8802ee9d0249 R09:
ffff8801a75da0ac
[199961.607240] R10: ffff8801a75db000 R11: ffff8802ee9d027d R12:
00000000ffffffea
[199961.608788] R13: ffff8804176e3830 R14: ffff8804176e3838 R15:
000000c42a6ac008
[199961.610315] FS:  00007fe1fa7fc700(0000) GS:ffff88042fc80000(0000)
knlGS:0000000000000000
[199961.611838] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[199961.613357] CR2: 000000c42a6ac008 CR3: 00000000a8411000 CR4:
00000000000006e0
[199961.614864] Stack:
[199961.616327]  00001000fffffffe ffffffff81823780 000000000014487b
ffffea00069d7680
[199961.617820]  0000000000000001 ffff880418614c00 ffffffff8112b768
ffffea00069d7680
[199961.619310]  ffff880418614c00 000000000014487b 00000000024200ca
ffff88011d834900
[199961.620799] Call Trace:
[199961.622250]  [<ffffffff8112b768>] ? __frontswap_load+0x68/0xc0
[199961.623689]  [<ffffffff8112666c>] ? swap_readpage+0x8c/0x120
[199961.625115]  [<ffffffff81126e61>] ? read_swap_cache_async+0x21/0x40
[199961.626545]  [<ffffffff81126f96>] ? swapin_readahead+0x116/0x1e0
[199961.627973]  [<ffffffff812b704e>] ? radix_tree_lookup_slot+0xe/0x20
[199961.629398]  [<ffffffff8111236f>] ? do_swap_page+0x42f/0x660
[199961.630799]  [<ffffffff81114bca>] ? handle_mm_fault+0x76a/0x1080
[199961.632163]  [<ffffffff811544ec>] ? new_sync_read+0xac/0xe0
[199961.633496]  [<ffffffff8102c7a9>] ? __do_page_fault+0x169/0x3e0
[199961.634798]  [<ffffffff8102ca5b>] ? do_page_fault+0x1b/0x60
[199961.636106]  [<ffffffff810e3cc9>] ?

Quoted 8 lines

__context_tracking_exit.part.1+0x49/0x60
[199961.637424]  [<ffffffff8157c7cf>] ? page_fault+0x1f/0x30
[199961.638739] Code: fb ff ff 41 c6 45 08 00 48 83 c4 08 44 89 e0 5b 5d 41 5c
41 5d 41 5e c3 be 0f 00 00 00 48 c7 c7 12 d6 6d 81 e8 e0 d2 f1 ff eb b0 <0f> 0b
0f 1f 84 00 00 00 00 00 66 2e 0f 1f 84 00 00 00 00 00 66 
[199961.641538] RIP  [<ffffffff8112c6c2>] zswap_frontswap_load+0x142/0x160
[199961.642922]  RSP <ffffc90000513cb0>
[199961.648971] ---[ end trace 76742a0cd4818a78 ]---

Alexandr

2017-01-18 05:59 UTC #4 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

D? D!N?, 18/01/2017 D2 10:39 +0900, Sergey Senozhatsky D?D,N?DuN?:

Quoted 25 lines

Cc Dan
Quoted 5 lines · Andrew Morton
Quoted 5 lines
https://bugzilla.kernel.org/show_bug.cgi?id=192571
--- Comment #1 from Gluzskiy Alexandr <[email protected]> ---
[199961.576604] ------------[ cut here ]------------
[199961.577830] kernel BUG at mm/zswap.c:1108!
zswap didn't manage to decompress the page:
static int zswap_frontswap_load(unsigned type, pgoff_t offset,
struct page *page)
{
...
	dst = kmap_atomic(page);
	tfm = *get_cpu_ptr(entry->pool->tfm);
	ret = crypto_comp_decompress(tfm, src, entry->length, dst,
&dlen);
	put_cpu_ptr(entry->pool->tfm);
	kunmap_atomic(dst);
	zpool_unmap_handle(entry->pool->zpool, entry->handle);
	BUG_ON(ret);
	^^^^^^^^^^^

Quoted 69 lines

is there anything suspicious in dmesg?

-ss

[..]

Quoted 64 lines

[199961.596459] Hardware name: System manufacturer System Product
Name/M4A77TD,
BIOS 2104A A A A 06/28/2010
[199961.597974] task: ffff880035c19680 task.stack:
ffffc90000510000
[199961.599490] RIP:
0010:[<ffffffff8112c6c2>]A A [<ffffffff8112c6c2>]
zswap_frontswap_load+0x142/0x160
[199961.601042] RSP: 0000:ffffc90000513cb0A A EFLAGS: 00010282
[199961.602588] RAX: ffffffff818263a0 RBX: ffff88036b2fb930 RCX:
ffffc90000513c98
[199961.604141] RDX: ffff8801a75da000 RSI: ffff8802ee9d0240 RDI:
ffff88041c25d000
[199961.605687] RBP: ffff880035c19680 R08: ffff8802ee9d0249 R09:
ffff8801a75da0ac
[199961.607240] R10: ffff8801a75db000 R11: ffff8802ee9d027d R12:
00000000ffffffea
[199961.608788] R13: ffff8804176e3830 R14: ffff8804176e3838 R15:
000000c42a6ac008
[199961.610315] FS:A A 00007fe1fa7fc700(0000)
GS:ffff88042fc80000(0000)
knlGS:0000000000000000
[199961.611838] CS:A A 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[199961.613357] CR2: 000000c42a6ac008 CR3: 00000000a8411000 CR4:
00000000000006e0
[199961.614864] Stack:
[199961.616327]A A 00001000fffffffe ffffffff81823780
000000000014487b
ffffea00069d7680
[199961.617820]A A 0000000000000001 ffff880418614c00
ffffffff8112b768
ffffea00069d7680
[199961.619310]A A ffff880418614c00 000000000014487b
00000000024200ca
ffff88011d834900
[199961.620799] Call Trace:
[199961.622250]A A [<ffffffff8112b768>] ?
__frontswap_load+0x68/0xc0
[199961.623689]A A [<ffffffff8112666c>] ? swap_readpage+0x8c/0x120
[199961.625115]A A [<ffffffff81126e61>] ?
read_swap_cache_async+0x21/0x40
[199961.626545]A A [<ffffffff81126f96>] ?
swapin_readahead+0x116/0x1e0
[199961.627973]A A [<ffffffff812b704e>] ?
radix_tree_lookup_slot+0xe/0x20
[199961.629398]A A [<ffffffff8111236f>] ? do_swap_page+0x42f/0x660
[199961.630799]A A [<ffffffff81114bca>] ?
handle_mm_fault+0x76a/0x1080
[199961.632163]A A [<ffffffff811544ec>] ? new_sync_read+0xac/0xe0
[199961.633496]A A [<ffffffff8102c7a9>] ?
__do_page_fault+0x169/0x3e0
[199961.634798]A A [<ffffffff8102ca5b>] ? do_page_fault+0x1b/0x60
[199961.636106]A A [<ffffffff810e3cc9>] ?
__context_tracking_exit.part.1+0x49/0x60
[199961.637424]A A [<ffffffff8157c7cf>] ? page_fault+0x1f/0x30
[199961.638739] Code: fb ff ff 41 c6 45 08 00 48 83 c4 08 44 89

e0 5b 5d 41 5c
41 5d 41 5e c3 be 0f 00 00 00 48 c7 c7 12 d6 6d 81 e8 e0 d2 f1 ff eb b0 <0f> 0b
0f 1f 84 00 00 00 00 00 66 2e 0f 1f 84 00 00 00 00 00 66A

[199961.641538] RIPA A [<ffffffff8112c6c2>]
zswap_frontswap_load+0x142/0x160
[199961.642922]A A RSP <ffffc90000513cb0>
[199961.648971] ---[ end trace 76742a0cd4818a78 ]---

no, nothing interesting in dmesg. but i suspect what it may be because of usage zram and zswap together.
i have following configuration:
1. boot option to kernel "ro radeon.audio=0 dma_debug=off reboot=warm gbpages rootfstype=ext4 rootflags=relatime,user_xattr,journal_async_commit,delalloc,nobarrier zswap.enabled=1"
2. zram activation in userspace:
"
cat /etc/local.d/zram.startA
#!/bin/sh

modprobe zram
echo 10G > /sys/block/zram0/disksize
mkswap /dev/zram0
swapon --priority 200 /dev/zram0
"

3. also i have normal swap block device as fall back if all memory used "
swapon
NAMEA A A A A A A A A A A A A TYPEA A A A A A SIZE USED PRIO /dev/mapper/swap partitionA A 16GA A A 0BA A A A 0 /dev/zram0A A A A A A A partitionA A 10GA A A 1GA A 200
"

so, maybe problem related to zswap on zram ? just a guess... -----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEl/sA7WQg6czXWI/dsEApXVthX7wFAlh/BBIACgkQsEApXVth X7w9lw/+O55XK/YZHszD/DMKRuZaaAQz7to/JrkJOCOJaYsV/PpUBh6liqYH8LCV 6vYaavzKt3ICW1qRa6Wjj7QC2YZKZTe8i8ERGTamDOnSu/gMlJz3EQ/uOEsNxde5 eoJr9n+JtUqf0PUUaMc61FcRbePcb3csQDD7KAwMSO7Q7+uP/osFUApjFVBOv0yd KggONcuyIlE0CIhmMk31Id+C7XoKeJogHa2qTIolGzi+yLCmiL+q+CujfXfrbOAz N6mDr7v6RTwzzOyXULZahceVxVtpUSgj84HG9wxTF7dwN6kwbW/YtdMu7UruqRyb SYHauUQSuEcbyb5m7tAPWfy4WsWaTacscdBCrOVqYJcn0nb945RMDz0RPIFZmLQS da6/zh67UF9KuSgprVakvgQ/ITJOfd96USlwZ+E8icJzT36IPWkSmFe6pNEa+KMn FiUf0JPN6ivO2q2wuwkIEKIeLiqDNX7QwcMxowMHKxezZobrzdyd4LoLx143mAa/ Ls0nABaN9bk+jzl3Ffl2Vx7YowuercwGaRzBuPEdxVQflA1gVPi7o/zwJ75CPAre ntQk8nWAqpxB30s0/++xYPbYaJFqWtXM2e4AQKQjiZSAdq34yl+q+di/1iGS/u4Q gfvGaprAtViK6AqURT8dXrWTv8KzAT2prIs3wdpmrc3V92p1cAo= =5ZmQ
-----END PGP SIGNATURE-----

Michal Hocko

2017-01-18 13:17 UTC #11 lore raw

In reply to Sergey Senozhatsky

Quoted 15 lines · Sergey Senozhatsky

Cc Dan
Quoted 5 lines · Andrew Morton
Quoted 5 lines
https://bugzilla.kernel.org/show_bug.cgi?id=192571
--- Comment #1 from Gluzskiy Alexandr <[email protected]> ---
[199961.576604] ------------[ cut here ]------------
[199961.577830] kernel BUG at mm/zswap.c:1108!
zswap didn't manage to decompress the page:
static int zswap_frontswap_load(unsigned type, pgoff_t offset,
struct page *page)
{
...

Quoted 8 lines

	dst = kmap_atomic(page);
	tfm = *get_cpu_ptr(entry->pool->tfm);
	ret = crypto_comp_decompress(tfm, src, entry->length, dst, &dlen);
	put_cpu_ptr(entry->pool->tfm);
	kunmap_atomic(dst);
	zpool_unmap_handle(entry->pool->zpool, entry->handle);
	BUG_ON(ret);
	^^^^^^^^^^^

Ugh, why do we even do that? This is not the way how to handle error situations. AFAIU propagating the error out wouldn't be a big deal because we would just fallback to regular swap, right?

Signature · 8 lines

Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Dan Streetman

2017-01-19 01:37 UTC #12 lore raw

Quoted 25 lines · Michal Hocko

Quoted 15 lines · Sergey Senozhatsky
Cc Dan
Quoted 5 lines · Andrew Morton
Quoted 5 lines
https://bugzilla.kernel.org/show_bug.cgi?id=192571
--- Comment #1 from Gluzskiy Alexandr <[email protected]> ---
[199961.576604] ------------[ cut here ]------------
[199961.577830] kernel BUG at mm/zswap.c:1108!
zswap didn't manage to decompress the page:
static int zswap_frontswap_load(unsigned type, pgoff_t offset,
struct page *page)
{
...
Quoted 8 lines
      dst = kmap_atomic(page);
      tfm = *get_cpu_ptr(entry->pool->tfm);
      ret = crypto_comp_decompress(tfm, src, entry->length, dst, &dlen);
      put_cpu_ptr(entry->pool->tfm);
      kunmap_atomic(dst);
      zpool_unmap_handle(entry->pool->zpool, entry->handle);
      BUG_ON(ret);
      ^^^^^^^^^^^

Quoted 3 lines

Ugh, why do we even do that? This is not the way how to handle error situations. AFAIU propagating the error out wouldn't be a big deal because we would just fallback to regular swap, right?

yeah this function definitely should never bug; it's just a callback from the zpool to try to write a page back to the swapcache so the zpool can free a page. It's definitely ok for it to return an error.

Quoted 10 lines

--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Dan Streetman

2017-01-19 02:57 UTC #5 lore raw

In reply to Alexandr

Quoted 12 lines · Alexandr

no, nothing interesting in dmesg. but i suspect what it may be because of usage zram and zswap together.
i have following configuration:
1. boot option to kernel "ro radeon.audio=0 dma_debug=off reboot=warm gbpages rootfstype=ext4 rootflags=relatime,user_xattr,journal_async_commit,delalloc,nobarrier zswap.enabled=1"
2. zram activation in userspace:
"
cat /etc/local.d/zram.start
#!/bin/sh

Quoted 4 lines

modprobe zram
echo 10G > /sys/block/zram0/disksize
mkswap /dev/zram0
swapon --priority 200 /dev/zram0

Why would you do this? There's no benefit of using zswap together with zram.

Quoted 11 lines

"
3. also i have normal swap block device as fall back if all memory used "
swapon
NAME TYPE SIZE USED PRIO
/dev/mapper/swap partition  16G   0B    0
/dev/zram0       partition  10G   1G  200
"
so, maybe problem related to zswap on zram ? just a guess...

it think it's unlikely, but it's hard to tell exactly why the page couldn't be uncompressed; my guess would be more likely a bug in the zpool backend. Were you using the default (zbud)?

Quoted 17 lines

-----BEGIN PGP SIGNATURE-----
iQIzBAEBCgAdFiEEl/sA7WQg6czXWI/dsEApXVthX7wFAlh/BBIACgkQsEApXVth X7w9lw/+O55XK/YZHszD/DMKRuZaaAQz7to/JrkJOCOJaYsV/PpUBh6liqYH8LCV 6vYaavzKt3ICW1qRa6Wjj7QC2YZKZTe8i8ERGTamDOnSu/gMlJz3EQ/uOEsNxde5 eoJr9n+JtUqf0PUUaMc61FcRbePcb3csQDD7KAwMSO7Q7+uP/osFUApjFVBOv0yd KggONcuyIlE0CIhmMk31Id+C7XoKeJogHa2qTIolGzi+yLCmiL+q+CujfXfrbOAz N6mDr7v6RTwzzOyXULZahceVxVtpUSgj84HG9wxTF7dwN6kwbW/YtdMu7UruqRyb SYHauUQSuEcbyb5m7tAPWfy4WsWaTacscdBCrOVqYJcn0nb945RMDz0RPIFZmLQS da6/zh67UF9KuSgprVakvgQ/ITJOfd96USlwZ+E8icJzT36IPWkSmFe6pNEa+KMn FiUf0JPN6ivO2q2wuwkIEKIeLiqDNX7QwcMxowMHKxezZobrzdyd4LoLx143mAa/ Ls0nABaN9bk+jzl3Ffl2Vx7YowuercwGaRzBuPEdxVQflA1gVPi7o/zwJ75CPAre ntQk8nWAqpxB30s0/++xYPbYaJFqWtXM2e4AQKQjiZSAdq34yl+q+di/1iGS/u4Q gfvGaprAtViK6AqURT8dXrWTv8KzAT2prIs3wdpmrc3V92p1cAo= =5ZmQ
-----END PGP SIGNATURE-----

Sergey Senozhatsky

2017-01-19 02:59 UTC #13 lore raw

In reply to Dan Streetman

Quoted 26 lines · Dan Streetman

Quoted 25 lines · Michal Hocko
Quoted 23 lines · Sergey Senozhatsky
Cc Dan
Quoted 5 lines · Andrew Morton
Quoted 5 lines
https://bugzilla.kernel.org/show_bug.cgi?id=192571
--- Comment #1 from Gluzskiy Alexandr <[email protected]> ---
[199961.576604] ------------[ cut here ]------------
[199961.577830] kernel BUG at mm/zswap.c:1108!
zswap didn't manage to decompress the page:
static int zswap_frontswap_load(unsigned type, pgoff_t offset,
struct page *page)
{
...
      dst = kmap_atomic(page);
      tfm = *get_cpu_ptr(entry->pool->tfm);
      ret = crypto_comp_decompress(tfm, src, entry->length, dst, &dlen);
      put_cpu_ptr(entry->pool->tfm);
      kunmap_atomic(dst);
      zpool_unmap_handle(entry->pool->zpool, entry->handle);
      BUG_ON(ret);
      ^^^^^^^^^^^

Quoted 8 lines

Quoted 3 lines
Ugh, why do we even do that? This is not the way how to handle error situations. AFAIU propagating the error out wouldn't be a big deal because we would just fallback to regular swap, right?
yeah this function definitely should never bug; it's just a callback from the zpool to try to write a page back to the swapcache so the zpool can free a page. It's definitely ok for it to return an error.

good. Dan, Seth, care to send the patch?
and one more thing... can you take a look at [1]?

[1] https://marc.info/?l=linux-mm&m=147031191906154&w=4

-ss

Dan Streetman

2017-01-19 04:21 UTC #14 lore raw

PATCH 1/1 zswap: change BUG to WARN in zswap_writeback_entry

Change the BUG calls to WARN, and return error.

There's no need to call BUG from this function, as it can safely return the error. The only caller of this function is the zpool that zswap is using, when zswap is trying to reduce the zpool size. While the error does indicate a bug, as none of the WARN conditions should ever happen, the zpool implementation can recover by trying to evict another page or zswap will recover by sending the new page to the swap disk.

This was reported in kernel bug 192571:
https://bugzilla.kernel.org/show_bug.cgi?id=192571

Reported-by Gluzskiy Alexandr <[email protected]>

Signed-off-by Dan Streetman <[email protected]>

---
 mm/zswap.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

mm/zswap.c +11 -3

index 067a0d6..60c4e6f 100644
@@ -787,7 +787,10 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle)
 		return 0;
 	}
 	spin_unlock(&tree->lock);
-	BUG_ON(offset != entry->offset);
+	if (WARN_ON(offset != entry->offset)) {
+		ret = -EINVAL;
+		goto fail;
+	}
 
 	/* try to allocate swap cache page */
 	switch (zswap_get_swap_cache_page(swpentry, &page)) {

@@ -813,8 +816,13 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle)
 		put_cpu_ptr(entry->pool->tfm);
 		kunmap_atomic(dst);
 		zpool_unmap_handle(entry->pool->zpool, entry->handle);
-		BUG_ON(ret);
-		BUG_ON(dlen != PAGE_SIZE);
+		if (WARN(ret, "error decompressing page: %d\n", ret))
+			goto fail;
+		if (WARN(dlen != PAGE_SIZE,
+			 "decompressed page only %x bytes\n", dlen)) {
+			ret = -EINVAL;
+			goto fail;
+		}
 
 		/* page is up to date */
 		SetPageUptodate(page);

Signature · 7 lines

2.9.3

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Dan Streetman

2017-01-19 04:43 UTC #17 lore raw

In reply to Sergey Senozhatsky

Quoted 31 lines · Sergey Senozhatsky

Quoted 30 lines · Dan Streetman
Quoted 28 lines · Michal Hocko
Quoted 15 lines · Sergey Senozhatsky
Cc Dan
Quoted 5 lines · Andrew Morton
Quoted 5 lines
https://bugzilla.kernel.org/show_bug.cgi?id=192571
--- Comment #1 from Gluzskiy Alexandr <[email protected]> ---
[199961.576604] ------------[ cut here ]------------
[199961.577830] kernel BUG at mm/zswap.c:1108!
zswap didn't manage to decompress the page:
static int zswap_frontswap_load(unsigned type, pgoff_t offset,
struct page *page)
{
...
Quoted 8 lines
      dst = kmap_atomic(page);
      tfm = *get_cpu_ptr(entry->pool->tfm);
      ret = crypto_comp_decompress(tfm, src, entry->length, dst, &dlen);
      put_cpu_ptr(entry->pool->tfm);
      kunmap_atomic(dst);
      zpool_unmap_handle(entry->pool->zpool, entry->handle);
      BUG_ON(ret);
      ^^^^^^^^^^^
Ugh, why do we even do that? This is not the way how to handle error situations. AFAIU propagating the error out wouldn't be a big deal because we would just fallback to regular swap, right?

Quoted 6 lines

Quoted 4 lines
yeah this function definitely should never bug; it's just a callback from the zpool to try to write a page back to the swapcache so the zpool can free a page. It's definitely ok for it to return an error.
good. Dan, Seth, care to send the patch?

damn, i misread the trace. I need to stop reading bug emails late at night.

I just sent a patch to change the BUG calls in zswap_writeback_entry() as those are totally recoverable, but the BUG here is from zswap_frontswap_load(), and a failure there isn't recoverable.

Unfortunately, if we accepted the page, but now can't recover it, it's gone and returning error from zswap_frontswap_load() will just cause page_io.c to go read the swap disk; and what's on there is not the page we're looking for, it's undefined as we didn't actually ever write the swap page to disk (unless frontswap_writethrough is enabled).

Ill have to look at this again in the morning.

Quoted 1 line

and one more thing... can you take a look at [1]?

yeah, i've been meaning to get to that as well, will send something tomorrow.

Quoted 5 lines

[1] https://marc.info/?l=linux-mm&m=147031191906154&w=4
        -ss

Sergey Senozhatsky

2017-01-19 05:14 UTC #15 lore raw

Re: [PATCH] zswap: change BUG to WARN in zswap_writeback_entry

In reply to Dan Streetman

Quoted 40 lines · Dan Streetman

Change the BUG calls to WARN, and return error.
There's no need to call BUG from this function, as it can safely return the error. The only caller of this function is the zpool that zswap is using, when zswap is trying to reduce the zpool size. While the error does indicate a bug, as none of the WARN conditions should ever happen, the zpool implementation can recover by trying to evict another page or zswap will recover by sending the new page to the swap disk.
This was reported in kernel bug 192571:
https://bugzilla.kernel.org/show_bug.cgi?id=192571
Reported-by Gluzskiy Alexandr <[email protected]>
Signed-off-by Dan Streetman <[email protected]>
---
 mm/zswap.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
mm/zswap.c +4 -3
index 067a0d6..60c4e6f 100644
@@ -787,7 +787,10 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle)
 		return 0;
 	}
 	spin_unlock(&tree->lock);
-	BUG_ON(offset != entry->offset);
+	if (WARN_ON(offset != entry->offset)) {
+		ret = -EINVAL;
+		goto fail;
+	}
 
 	/* try to allocate swap cache page */
 	switch (zswap_get_swap_cache_page(swpentry, &page)) {
@@ -813,8 +816,13 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle)
 		put_cpu_ptr(entry->pool->tfm);
 		kunmap_atomic(dst);
 		zpool_unmap_handle(entry->pool->zpool, entry->handle);
-		BUG_ON(ret);
-		BUG_ON(dlen != PAGE_SIZE);

mm/zswap.c static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) · line 828

Quoted 10 lines

mm/zswap.c +7 -0

+		if (WARN(ret, "error decompressing page: %d\n", ret))
+			goto fail;
+		if (WARN(dlen != PAGE_SIZE,
+			 "decompressed page only %x bytes\n", dlen)) {
+			ret = -EINVAL;
+			goto fail;
+		}
 
 		/* page is up to date */
 		SetPageUptodate(page);

+ zswap_frontswap_load() I suppose.

mm/zswap.c +2 -3

index 067a0d62f318..e2743687a202 100644
@@ -1023,13 +1023,13 @@ static int zswap_frontswap_load(unsigned type, pgoff_t offset,
        put_cpu_ptr(entry->pool->tfm);
        kunmap_atomic(dst);
        zpool_unmap_handle(entry->pool->zpool, entry->handle);
-       BUG_ON(ret);
+       WARN(ret, "error decompressing page: %d\n", ret);
 
        spin_lock(&tree->lock);
        zswap_entry_put(tree, entry);
        spin_unlock(&tree->lock);
 
-       return 0;
+       return ret;
 }
 
--

To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Dan Streetman

2017-01-19 15:37 UTC #16 lore raw

Re: [PATCH] zswap: change BUG to WARN in zswap_writeback_entry

Quoted 10 lines · Sergey Senozhatsky

Quoted 9 lines · Dan Streetman
Change the BUG calls to WARN, and return error.
There's no need to call BUG from this function, as it can safely return the error. The only caller of this function is the zpool that zswap is using, when zswap is trying to reduce the zpool size. While the error does indicate a bug, as none of the WARN conditions should ever happen, the zpool implementation can recover by trying to evict another page or zswap will recover by sending the new page to the swap disk.

Quoted 2 lines

Quoted 2 lines
This was reported in kernel bug 192571:
https://bugzilla.kernel.org/show_bug.cgi?id=192571

Andrew, please ignore this patch (for now at least)...it won't address this bug.

Quoted 37 lines

Reported-by Gluzskiy Alexandr <[email protected]>

Signed-off-by Dan Streetman <[email protected]>

---
 mm/zswap.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

mm/zswap.c +11 -3

index 067a0d6..60c4e6f 100644
@@ -787,7 +787,10 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle)
              return 0;
      }
      spin_unlock(&tree->lock);
-     BUG_ON(offset != entry->offset);
+     if (WARN_ON(offset != entry->offset)) {
+             ret = -EINVAL;
+             goto fail;
+     }
 
      /* try to allocate swap cache page */
      switch (zswap_get_swap_cache_page(swpentry, &page)) {

@@ -813,8 +816,13 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle)
              put_cpu_ptr(entry->pool->tfm);
              kunmap_atomic(dst);
              zpool_unmap_handle(entry->pool->zpool, entry->handle);
-             BUG_ON(ret);
-             BUG_ON(dlen != PAGE_SIZE);
+             if (WARN(ret, "error decompressing page: %d\n", ret))
+                     goto fail;
+             if (WARN(dlen != PAGE_SIZE,
+                      "decompressed page only %x bytes\n", dlen)) {
+                     ret = -EINVAL;
+                     goto fail;
+             }

Quoted 5 lines

mm/zswap.c static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) · line 828
Quoted 2 lines
mm/zswap.c +0 -0
              /* page is up to date */
              SetPageUptodate(page);
+ zswap_frontswap_load() I suppose.

So my initial comment before that it was safe to switch to WARN was right for zswap_writeback_entry(), but not right for zswap_frontswap_load() - it was late and I didn't read the trace correctly :(

At the BUG point in zswap_frontswap_load(), we have found the page in the rb tree (via zswap_entry_find_get) so we know that we did accept the page for storage, so we're the only place who has a copy of it (assuming frontswap_writethrough isn't enabled). If we can't decompress it, then we only have 2 choices - BUG or return error. If we return error, frontswap will try any other frontswap backends it has registered (none, I assume, or if so they should not have this swap offset's entry, since we found a match). After frontswap can't recover the page from any of its backends, it will return error, and swap_readpage() will then assume the page is actually stored on the swap disk, and read it back. However, the page wasn't written to the swap disk (since we stored it in zswap), and whatever is read back from the disk is not what was originally in the page - leading to memory corruption. So it's better to BUG at this point.

As to why it can't decompress the page, we should at least add info about the error value. Maybe the zpool storage encountered a bug and provided the wrong compressed data to us...

Quoted 10 lines

mm/zswap.c +0 -1

index 067a0d62f318..e2743687a202 100644
@@ -1023,13 +1023,13 @@ static int zswap_frontswap_load(unsigned type, pgoff_t offset,
        put_cpu_ptr(entry->pool->tfm);
        kunmap_atomic(dst);
        zpool_unmap_handle(entry->pool->zpool, entry->handle);
-       BUG_ON(ret);

mm/zswap.c static int zswap_frontswap_load(unsigned type, pgoff_t offset, · line 1034

Quoted 10 lines

mm/zswap.c +2 -1

+       WARN(ret, "error decompressing page: %d\n", ret);
 
        spin_lock(&tree->lock);
        zswap_entry_put(tree, entry);
        spin_unlock(&tree->lock);
 
-       return 0;
+       return ret;
 }

Alexandr

2017-01-24 00:03 UTC #6 lore raw

In reply to Dan Streetman

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Quoted 2 lines

Why would you do this?A A There's no benefit of using zswap together with zram.

i just wanted to test zram and zswap, i still not dig to deep in it, but what i wanted is to use zram swap (with zswap disabled), and if it exceeded use real swap on block device with zswap enabled. -----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEl/sA7WQg6czXWI/dsEApXVthX7wFAliGmbkACgkQsEApXVth X7zyDQ/9HMnJ5JzAAkrWJKlvpA+H6CRw0YBO77zQ44lr9R5jqmVAhU3XS6+dfYpA ZL9lwG8zEqSDUCakko4vRVaeOiy3qzCNQcect2J1I9aGrHFIkC0I/ifPpbXRa4s5 +D45mSUzGxnMMz1XZrOkvuNsbzdWuTmQqTUqnJVovRD/V62u8Y50gDL3zkz/9x7L mLjl/5WGjBAOQtwYpq1uE7FAJFHjV2cX8yI5JrFzMK1oghjFfqPFiYbD0yqSR2MB QFdDQlqhMZ7Dwnk0P/WzIpJXdoT2NXH1iWRsvvKYeMwRP7hIzEnkpxfTlYtBK5xu 7zw/IEa0prLaEtYEh1j6h8Tzn6wKNeIT3t0g2yBT3QC8BW/v7AODlj95C+jIR06f tikDCx+DUDuP96SW6RIjVLODCt/4yCzgVdxoAD5AbAyY+pU+JEmDkz8L60Gk2mC9 OG9IExiCCY/G3069A6UZROSFrrZGgrP75JGhTP91cS/XGH/HODFmqHQVVp45cED9 wn820IGjB2AAI6MmmRCvgqzUs99PTv8Xqr/x2Ea/ce+lFiU+L5x+xY7Q1q3KhQpQ pLqLShi9iQUAzIYAtXZNPlbgwDtbYqz5sIAa6cmiv92bcRgJdPf4SiWLBUzIVi0M KkNXiyo3XkDXoC8P1WjzLoexoJtOJooUbPcKimCI8Ef6+s5PmC0= =Pyde
-----END PGP SIGNATURE-----

Dan Streetman

2017-01-24 20:17 UTC #7 lore raw

Quoted 10 lines · Alexandr

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Quoted 2 lines
Why would you do this? There's no benefit of using zswap together with zram.
i just wanted to test zram and zswap, i still not dig to deep in it, but what i wanted is to use zram swap (with zswap disabled), and if it exceeded use real swap on block device with zswap enabled.

I don't believe that's possible, you can't enable zswap for only specific swap devices; and anyway, if you fill up zram, you won't really have any memory left for zswap to use will you?

However, it shouldn't encounter any BUG(), like you saw. If it's reproducable for you, can you give details on how to reproduce it?

Chulmin Kim

2017-01-25 04:02 UTC #8 lore raw

Quoted 12 lines · Dan Streetman

Quoted 10 lines · Alexandr
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Quoted 2 lines
Why would you do this? There's no benefit of using zswap together with zram.
i just wanted to test zram and zswap, i still not dig to deep in it, but what i wanted is to use zram swap (with zswap disabled), and if it exceeded use real swap on block device with zswap enabled.

Quoted 7 lines

I don't believe that's possible, you can't enable zswap for only specific swap devices; and anyway, if you fill up zram, you won't really have any memory left for zswap to use will you?
However, it shouldn't encounter any BUG(), like you saw. If it's reproducable for you, can you give details on how to reproduce it?

Hello. Mr. Streetman.

Regarding to this problem, I have a question on zswap.

Is there any reason that
zswap_frontswap_load() does not call flush_dcache_page()?

The zswap load function can dirty the page mapped to user space (might be shareable/writable) which seems exactly the condition mentioned in the definition of flush_dcache_page().

I'm thinking that
flush_dcache_page() should be called in the end of zswap_frontswap_load(). Could you review my opinion?

Thanks!
Chulmin Kim

Quoted 7 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Alexandr

2017-01-25 06:42 UTC #10 lore raw

In reply to Dan Streetman

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

D? D?N?, 24/01/2017 D2 15:16 -0500, Dan Streetman D?D,N?DuN?:

Quoted 15 lines · Alexandr

Quoted 8 lines
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Quoted 3 lines
Why would you do this?A A There's no benefit of using zswap together
with zram.
Quoted 5 lines
i just wanted to test zram and zswap, i still not dig to deep in it,
but what i wanted is to use zram swap (with zswap disabled), and if it
exceeded use real swap on block device with zswap enabled.

Quoted 6 lines

I don't believe that's possible, you can't enable zswap for only specific swap devices; and anyway, if you fill up zram, you won't really have any memory left for zswap to use will you?
However, it shouldn't encounter any BUG(), like you saw.A A If it's reproducable for you, can you give details on how to reproduce it?

it happened only once, and i am noticed it only because few applications hang, but i can run this setup for a while, and let know if it happen again, it happened on io and memory intensive app, this machine have heavy load sometime, so i think it may happen again. -----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEl/sA7WQg6czXWI/dsEApXVthX7wFAliISMcACgkQsEApXVth X7zgMA/+KoI1rdpCfJdxrihlkKavJcfYR/EoI4FGzQadJb6mZSihzuHcLVcIhiLV VH+9HNADgygur//EQMAsliqT7HNxdEIpouMU/4w9dDxWiUFaAFo6kQYztVSQog8X Kd3zJ1YagxSOXv0yx/OiR40/NwXygLSW8zRQ0rVwOIO6TF05lJYUA5QQ6F+izHGB syNDQUwOukQ8Bcaxctic+uE/nn55ufkHMyjCtlQG2jG6/gk1590fzxugsk69U0Ou qq8zFyShhYQ2onw36cJWi62rXpKvj7mj/suo7FwwmmLBS2R9jcrQILTnYnhAM+YH JkVsIjXJrIWGLd3jeFpHwJMmvuMe5jPT3ppGGx3m4QbdRe+DAujT+5bWaQC5ubnN 4H84h6kGEsNTelf2rfZs58MomQy61adgSwKqMpOw81b/H11fYuZTVmlqBkyFKzos 0fkSTdkpHXSoKkLw6sgr2ch+jLJanR29+T9VRuR2m4+PRdLrUZF3L5HBejYDkE5O 3eF+eR/cVXoyZleVUAJaG7KAM+P8KEvz5kZAOOTGixFM23L1KnIRejYcjpYgKGkG Q4k5+w56ONkzmL6IKqx5eOHstCxSl1R/uKNNN9rwrq1sRuRUpcQlxFfneVS7U9eo pKYcoyO/yiYKdXTH82d/LJBf6yISZcwMsBSPciSWXQuLPtvkeNE= =PnJw
-----END PGP SIGNATURE-----

Dan Streetman

2017-01-26 17:09 UTC #9 lore raw

In reply to Chulmin Kim

Quoted 33 lines · Chulmin Kim

Quoted 16 lines · Dan Streetman
Quoted 5 lines · Alexandr
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Quoted 7 lines
Quoted 2 lines
Why would you do this? There's no benefit of using zswap together with zram.
i just wanted to test zram and zswap, i still not dig to deep in it, but what i wanted is to use zram swap (with zswap disabled), and if it exceeded use real swap on block device with zswap enabled.
Quoted 7 lines
I don't believe that's possible, you can't enable zswap for only specific swap devices; and anyway, if you fill up zram, you won't really have any memory left for zswap to use will you?
However, it shouldn't encounter any BUG(), like you saw. If it's reproducable for you, can you give details on how to reproduce it?
Hello. Mr. Streetman.
Regarding to this problem, I have a question on zswap.
Is there any reason that
zswap_frontswap_load() does not call flush_dcache_page()?

Quoted 7 lines

The zswap load function can dirty the page mapped to user space (might be shareable/writable) which seems exactly the condition mentioned in the definition of flush_dcache_page().
I'm thinking that
flush_dcache_page() should be called in the end of zswap_frontswap_load(). Could you review my opinion?

I don't think it needs to, as i detailed in my response to the other thread.

Also, this is a different issue, I think - even if there is a cache problem with pages loaded from zswap, i don't see how it would cause a decompression failure - the zpool storage is the only code that has a copy of its compressed pages, no userspace or any other kernel code should be accessing any of it.

Quoted 20 lines

Thanks!
Chulmin Kim
Quoted 7 lines
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Quoted 6 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>