Skip to contentSkip to conversation

OOPS: linux-4.9.33/fs/pnode.c:propagate_one()

DISCUSSION Philipp Hahn 2 participants 4 messages Sep 28, 2017 – Oct 4, 2017 VFS

lore
Share & tools
lore
#0 lore raw

Hello Eric,

we have observed the following OOPS with linux-4.9.33 on several occasions when starting a Docker container:

Quoted 22 lines
[  531.801537] Oops: 0000 [#1] SMP
[  531.801565] Modules linked in: xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo xt_addrtype br_netfilter bridge stp llc overlay ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_tcpudp nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_mangle ip6table_filter ip6_tables xt_conntrack iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables x_tables rpcsec_gss_krb5 nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc joydev pcspkr serio_raw virtio_balloon i2c_piix4 evdev quota_v2 quota_tree parport_pc ppdev lp parport autofs4 ext4 crc16 jbd2 fscrypto mbcache dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_mod hid_generic usbhid hid ata_generic cirrus virtio_net virtio_blk ttm ata_piix libata uhci_hcd ehci_hcd psmouseSep 28 13:30:33 master70 kernel: [  531.802171]  drm_kms_helper scsi_mod floppy usbcore virtio_pci button drm virtio_ring virtio usb_common
[  531.802247] CPU: 0 PID: 1668 Comm: dockerd Not tainted 4.9.0-ucs104-amd64 #1 Debian 4.9.30-2A~4.2.0.201706171152
[  531.802310] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
[  531.802347] task: ffff918af7ccaf40 task.stack: ffffb47a40b54000
[  531.802385] RIP: 0010:[<ffffffff91234020>]  [<ffffffff91234020>] propagate_one+0x160/0x1e0
[  531.802444] RSP: 0018:ffffb47a40b57e08  EFLAGS: 00010282
[  531.802479] RAX: 0000000000000001 RBX: ffff918adf43cd80 RCX: 0000000000000000
[  531.802524] RDX: ffff918ad69209c0 RSI: ffff918adf43cd80 RDI: ffff918ad69209c0
[  531.802569] RBP: ffff918afb3a2480 R08: 0000000000000001 R09: ffff918ac0183e40
[  531.802614] R10: 00000000000011b8 R11: 00000000000176b9 R12: ffffb47a40b57e58
[  531.802670] R13: 0000000000000000 R14: ffff918adf43cd80 R15: ffff918afbe83600
[  531.802716] FS:  00007f6157fff700(0000) GS:ffff918affc00000(0000) knlGS:0000000000000000
[  531.802767] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  531.802804] CR2: 0000000000000010 CR3: 0000000039e60000 CR4: 00000000000006f0
[  531.802852] Stack:
[  531.802868]  ffff918adf43cd80 ffff918afb3a2480 ffffffff912344af ffff918ac0183e40
[  531.802923]  ffff918afbe83600 ffff918afb3a2480 0000000000000000 0000000000000000
[  531.802978]  ffffffff91226971 ffff918afe084900 ffff918ac01c4c40 3aa437d5d9ae5224
[  531.803037] Call Trace:
[  531.803062]  [<ffffffff912344af>] ? propagate_mnt+0x11f/0x150
[  531.803104]  [<ffffffff91226971>] ? attach_recursive_mnt+0x1d1/0x2f0
Quoted 8 lines
[  531.803146]  [<ffffffff912279cd>] ? do_mount+0xafd/0xc70
[  531.803182]  [<ffffffff91227e24>] ? SyS_mount+0x84/0xc0
[  531.803229]  [<ffffffff91611abb>] ? system_call_fast_compare_end+0xc/0x9b
[  531.803273] Code: 48 89 de eb 09 f6 42 33 04 75 7d 48 89 d6 48 8b 96 d8 00 00 00 48 39 fa 75 eb 48 8b 0d 52 d6 cc 00 4c 8b 0d 53 d6 cc 00 49 39 c9 <48> 8b 51 10 74 5c 48 3b ba d8 00 00 00 75 3f 45 84 c0 75 55 8b
[  531.805256] RIP  [<ffffffff91234020>] propagate_one+0x160/0x1e0
[  531.807057]  RSP <ffffb47a40b57e08>
[  531.808839] CR2: 0000000000000010
[  531.810618] ---[ end trace f817db65f78e36fc ]---

The same has happened in Amazon-EC2 and on out own KVM servers.

There was a similar bug report in 2016 at
<https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1572316>, but that one got fixed with "v4.6-rc7~24^2".

Maybe is has already been fixed by

Quoted 4 lines
$ git l1 v4.9.33..v4.9.52 -- fs/pnode.c
54fcb2303ef4 mnt: Make propagate_umount less slow for overlapping mount propagation trees
bb4fbf094b44 mnt: In propgate_umount handle visiting mounts in any order
e260db757676 mnt: In umount propagation reparent in a separate pass

but I'd like to ask anyway if you have seen such a OOPs before?

For your convenience here's the disassembly, where I think "rcx = last_source->mnt_parent = NULL":

Quoted 13 lines
		do {
			struct mount *parent = last_source->mnt_parent;
			if (last_source == first_source)
000000000000029d <propagate_one+0x15d> cmp    %rcx,%r9
			p = n->mnt_master;
			if (p == dest_master || IS_MNT_MARKED(p))
				break;
		}
		do {
			struct mount *parent = last_source->mnt_parent;
00000000000002a0 <propagate_one+0x160> mov    0x10(%rcx),%rdx
			if (last_source == first_source)

00000000000002a4 <propagate_one+0x164> je 0000000000000302 <propagate_one+0x1c2>

Quoted 8 lines
				break;
			done = parent->mnt_master == p;
			if (done && peers(n, parent))
00000000000002a6 <propagate_one+0x166> cmp    0xd8(%rdx),%rdi
00000000000002ad <propagate_one+0x16d> jne    00000000000002ee <propagate_one+0x1ae>
00000000000002af <propagate_one+0x16f> test   %r8b,%r8b
00000000000002b2 <propagate_one+0x172> jne    0000000000000309 <propagate_one+0x1c9>
00000000000002b4 <propagate_one+0x174> mov    0x110(%rsi),%eax

We will try 4.9.52 next and see, if we can still reproduce it.

Philipp

Signature · 15 lines
Philipp Hahn
Open Source Software Engineer

Univention GmbH
be open.
Mary-Somerville-Str. 1
D-28359 Bremen
Tel.: +49 421 22232-0
Fax : +49 421 22232-99
[email protected]

http://www.univention.de/
Geschäftsführer: Peter H. Ganten
HRB 20755 Amtsgericht Bremen
Steuer-Nr.: 71-597-02876
#1 lore raw
Quoted 42 lines · Philipp Hahn

Hello Eric,

we have observed the following OOPS with linux-4.9.33 on several occasions when starting a Docker container:

Quoted 22 lines
[  531.801537] Oops: 0000 [#1] SMP
[  531.801565] Modules linked in: xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo xt_addrtype br_netfilter bridge stp llc overlay ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_tcpudp nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_mangle ip6table_filter ip6_tables xt_conntrack iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables x_tables rpcsec_gss_krb5 nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc joydev pcspkr serio_raw virtio_balloon i2c_piix4 evdev quota_v2 quota_tree parport_pc ppdev lp parport autofs4 ext4 crc16 jbd2 fscrypto mbcache dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_mod hid_generic usbhid hid ata_generic cirrus virtio_net virtio_blk ttm ata_piix libata uhci_hcd ehci_hcd psmouseSep 28 13:30:33 master70 kernel: [  531.802171]  drm_kms_helper scsi_mod floppy usbcore virtio_pci button drm virtio_ring virtio usb_common
[  531.802247] CPU: 0 PID: 1668 Comm: dockerd Not tainted 4.9.0-ucs104-amd64 #1 Debian 4.9.30-2A~4.2.0.201706171152
[  531.802310] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
[  531.802347] task: ffff918af7ccaf40 task.stack: ffffb47a40b54000
[  531.802385] RIP: 0010:[<ffffffff91234020>]  [<ffffffff91234020>] propagate_one+0x160/0x1e0
[  531.802444] RSP: 0018:ffffb47a40b57e08  EFLAGS: 00010282
[  531.802479] RAX: 0000000000000001 RBX: ffff918adf43cd80 RCX: 0000000000000000
[  531.802524] RDX: ffff918ad69209c0 RSI: ffff918adf43cd80 RDI: ffff918ad69209c0
[  531.802569] RBP: ffff918afb3a2480 R08: 0000000000000001 R09: ffff918ac0183e40
[  531.802614] R10: 00000000000011b8 R11: 00000000000176b9 R12: ffffb47a40b57e58
[  531.802670] R13: 0000000000000000 R14: ffff918adf43cd80 R15: ffff918afbe83600
[  531.802716] FS:  00007f6157fff700(0000) GS:ffff918affc00000(0000) knlGS:0000000000000000
[  531.802767] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  531.802804] CR2: 0000000000000010 CR3: 0000000039e60000 CR4: 00000000000006f0
[  531.802852] Stack:
[  531.802868]  ffff918adf43cd80 ffff918afb3a2480 ffffffff912344af ffff918ac0183e40
[  531.802923]  ffff918afbe83600 ffff918afb3a2480 0000000000000000 0000000000000000
[  531.802978]  ffffffff91226971 ffff918afe084900 ffff918ac01c4c40 3aa437d5d9ae5224
[  531.803037] Call Trace:
[  531.803062]  [<ffffffff912344af>] ? propagate_mnt+0x11f/0x150
[  531.803104]  [<ffffffff91226971>] ? attach_recursive_mnt+0x1d1/0x2f0
Quoted 8 lines
[  531.803146]  [<ffffffff912279cd>] ? do_mount+0xafd/0xc70
[  531.803182]  [<ffffffff91227e24>] ? SyS_mount+0x84/0xc0
[  531.803229]  [<ffffffff91611abb>] ? system_call_fast_compare_end+0xc/0x9b
[  531.803273] Code: 48 89 de eb 09 f6 42 33 04 75 7d 48 89 d6 48 8b 96 d8 00 00 00 48 39 fa 75 eb 48 8b 0d 52 d6 cc 00 4c 8b 0d 53 d6 cc 00 49 39 c9 <48> 8b 51 10 74 5c 48 3b ba d8 00 00 00 75 3f 45 84 c0 75 55 8b
[  531.805256] RIP  [<ffffffff91234020>] propagate_one+0x160/0x1e0
[  531.807057]  RSP <ffffb47a40b57e08>
[  531.808839] CR2: 0000000000000010
[  531.810618] ---[ end trace f817db65f78e36fc ]---

The same has happened in Amazon-EC2 and on out own KVM servers.

There was a similar bug report in 2016 at
<https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1572316>, but that one got fixed with "v4.6-rc7~24^2".

Quoted 7 lines

Maybe is has already been fixed by

Quoted 4 lines
$ git l1 v4.9.33..v4.9.52 -- fs/pnode.c
54fcb2303ef4 mnt: Make propagate_umount less slow for overlapping mount propagation trees
bb4fbf094b44 mnt: In propgate_umount handle visiting mounts in any order
e260db757676 mnt: In umount propagation reparent in a separate pass

but I'd like to ask anyway if you have seen such a OOPs before?

I have not.

Quoted 2 lines

For your convenience here's the disassembly, where I think "rcx = last_source->mnt_parent = NULL":

If your disassembly is lined up properly with the code my guess would be that rcx simply holds the value of last_source. And rdx is being set to last_source->mnt_parent.

Disassembly a recent build on my machine I have a one byte difference from yours, but that is what I see happening. rcx == last_source.

Why and how last_source is getting set to NULL is something I need to read the code for a while to figure out.

Quoted 22 lines
Quoted 13 lines
		do {
			struct mount *parent = last_source->mnt_parent;
			if (last_source == first_source)
000000000000029d <propagate_one+0x15d> cmp    %rcx,%r9
			p = n->mnt_master;
			if (p == dest_master || IS_MNT_MARKED(p))
				break;
		}
		do {
			struct mount *parent = last_source->mnt_parent;
00000000000002a0 <propagate_one+0x160> mov    0x10(%rcx),%rdx
			if (last_source == first_source)

00000000000002a4 <propagate_one+0x164> je 0000000000000302 <propagate_one+0x1c2>

Quoted 8 lines
				break;
			done = parent->mnt_master == p;
			if (done && peers(n, parent))
00000000000002a6 <propagate_one+0x166> cmp    0xd8(%rdx),%rdi
00000000000002ad <propagate_one+0x16d> jne    00000000000002ee <propagate_one+0x1ae>
00000000000002af <propagate_one+0x16f> test   %r8b,%r8b
00000000000002b2 <propagate_one+0x172> jne    0000000000000309 <propagate_one+0x1c9>
00000000000002b4 <propagate_one+0x174> mov    0x110(%rsi),%eax
Quoted 1 line

We will try 4.9.52 next and see, if we can still reproduce it.

I suspect you will be able to as nothing as nothing substantial has changed in that logic in a while.

I will stare at it and see if I can imagine how you are getting last_source set to NULL.

Eric

#2 lore raw
Quoted 3 lines · [email protected] (Eric W. Biederman) writes:
Quoted 1 line · Philipp Hahn

We will try 4.9.52 next and see, if we can still reproduce it.

Can you still reproduce this on 4.9.52?

I am not seeing anything obvious that would result in last_source becomming NULL there so knowing I am not lookning for a phantom would be very helpful.

Eric

#3 lore raw

Hello Eric,

Am 03.10.2017 um 18:48 schrieb Eric W. Biederman:

Quoted 7 lines
Quoted 3 lines · [email protected] (Eric W. Biederman) writes:
Quoted 1 line · Philipp Hahn

We will try 4.9.52 next and see, if we can still reproduce it.

Can you still reproduce this on 4.9.52?

My college Dirk is able to reproduce it only with .33, but not with .52 (yet).

Quoted 3 lines

I am not seeing anything obvious that would result in last_source becomming NULL there so knowing I am not lookning for a phantom would be very helpful.

Looks like .52 closed the race windows sufficiently enough, so the bug does not get triggered there.

The bug is triggered when using "docker cp" to copy a file into the container. It communicates with the long-running 'dockerd', which rebuilds the overlay FS itself to access the file. The failing syscall is:

Quoted 1 line

mount("/fw/data_folder", "/var/lib/docker/overlay/038e54bc5db6a180409aa0e5b0f84087b93ac7ffe4eaa3c8908aea79d43a45d3/merged/fwxserver/Data Folder", 0xc82135e5c8, MS_BIND|MS_REC, NULL

I just now noticed that before the mount() docker did this:

Quoted 13 lines

umount("/var/lib/docker/overlay/038e54bc5db6a180409aa0e5b0f84087b93ac7ffe4eaa3c8908aea79d43a45d3/merged/usr/local/filewave/media", MNT_DETACH) = 0 umount("/var/lib/docker/overlay/038e54bc5db6a180409aa0e5b0f84087b93ac7ffe4eaa3c8908aea79d43a45d3/merged/usr/local/filewave/tmp", MNT_DETACH) = 0 umount("/var/lib/docker/overlay/038e54bc5db6a180409aa0e5b0f84087b93ac7ffe4eaa3c8908aea79d43a45d3/merged/usr/local/etc", MNT_DETACH) = 0 umount("/var/lib/docker/overlay/038e54bc5db6a180409aa0e5b0f84087b93ac7ffe4eaa3c8908aea79d43a45d3/merged/usr/local/filewave/log", MNT_DETACH) = 0 umount("/var/lib/docker/overlay/038e54bc5db6a180409aa0e5b0f84087b93ac7ffe4eaa3c8908aea79d43a45d3/merged/var/lib/univention-appcenter/apps/filewave/data", MNT_DETACH) = 0 umount("/var/lib/docker/overlay/038e54bc5db6a180409aa0e5b0f84087b93ac7ffe4eaa3c8908aea79d43a45d3/merged/var/lib/univention-appcenter/apps/filewave/conf", MNT_DETACH) = 0 umount("/var/lib/docker/overlay/038e54bc5db6a180409aa0e5b0f84087b93ac7ffe4eaa3c8908aea79d43a45d3/merged/usr/local/filewave/certs", MNT_DETACH) = 0 umount("/var/lib/docker/overlay/038e54bc5db6a180409aa0e5b0f84087b93ac7ffe4eaa3c8908aea79d43a45d3/merged/usr/local/filewave/fwcld", MNT_DETACH) = 0 umount("/var/lib/docker/overlay/038e54bc5db6a180409aa0e5b0f84087b93ac7ffe4eaa3c8908aea79d43a45d3/merged/fwxserver/Data Folder", MNT_DETACH) = 0 umount("/var/lib/docker/overlay/038e54bc5db6a180409aa0e5b0f84087b93ac7ffe4eaa3c8908aea79d43a45d3/merged/usr/local/filewave/apache/conf", MNT_DETACH) = 0 umount("/var/lib/docker/overlay/038e54bc5db6a180409aa0e5b0f84087b93ac7ffe4eaa3c8908aea79d43a45d3/merged/fwxserver/DB", MNT_DETACH) = 0 umount("/var/lib/docker/overlay/038e54bc5db6a180409aa0e5b0f84087b93ac7ffe4eaa3c8908aea79d43a45d3/merged/usr/local/filewave/apache/logs", MNT_DETACH) = 0 umount("/var/lib/docker/overlay/038e54bc5db6a180409aa0e5b0f84087b93ac7ffe4eaa3c8908aea79d43a45d3/merged/private/var/log", MNT_DETACH) = 0

Quoted 8 lines

umount("/var/lib/docker/overlay/038e54bc5db6a180409aa0e5b0f84087b93ac7ffe4eaa3c8908aea79d43a45d3/merged/usr/local/filewave/apache/passwords", MNT_DETACH) = 0 umount("/var/lib/docker/overlay/038e54bc5db6a180409aa0e5b0f84087b93ac7ffe4eaa3c8908aea79d43a45d3/merged/sys/fs/cgroup", MNT_DETACH) = 0 umount("/var/lib/docker/overlay/038e54bc5db6a180409aa0e5b0f84087b93ac7ffe4eaa3c8908aea79d43a45d3/merged/usr/local/filewave/ipa", MNT_DETACH) = 0 umount("/var/lib/docker/overlay/038e54bc5db6a180409aa0e5b0f84087b93ac7ffe4eaa3c8908aea79d43a45d3/merged/etc/apt/apt.conf.d/80proxy", MNT_DETACH) = 0 umount("/var/lib/docker/overlay/038e54bc5db6a180409aa0e5b0f84087b93ac7ffe4eaa3c8908aea79d43a45d3/merged/usr/local/filewave/postgresql/conf", MNT_DETACH) = 0 umount("/var/lib/docker/overlay/038e54bc5db6a180409aa0e5b0f84087b93ac7ffe4eaa3c8908aea79d43a45d3/merged/etc/resolv.conf", MNT_DETACH) = 0 umount("/var/lib/docker/overlay/038e54bc5db6a180409aa0e5b0f84087b93ac7ffe4eaa3c8908aea79d43a45d3/merged/etc/hostname", MNT_DETACH) = 0 umount("/var/lib/docker/overlay/038e54bc5db6a180409aa0e5b0f84087b93ac7ffe4eaa3c8908aea79d43a45d3/merged/etc/hosts", MNT_DETACH) = 0

Is MNT_DETACH a problem, as it is described to do a "lazy" umount? Doing another

Quoted 1 line

strace -ttt -b execve -f -ff -s 65536 -o /var/tmp/docker.strace -p $(</run/docker.pid)

shows the mount() following the umounts() < 0.2s, so maybe that's the missing detail.

If you need more data, just ask Dirk: he's able to re-produce the bug every time.

If you need more data from the kernel, I can build a customized kernel with more debug output if that helps.

Philipp