found a scenario for BUG at fs/ext4/super.c:804!
Subsystems (2)
With kernel 3.10-rcX there's a big likelihood to observe that issue if I do the following steps:
1. create a 257 MB file /mnt/ramdisk/disk0
2. create an EXT4 fs onto it
3. mount it onto /mnt/ramdisk/victims/
4. create files and directories in /mnt/ramdisk/victims/v1/v2
5. exportfs the directory /mnt/ramdisk/victims/ via NFS
6. start a user mode linux
7. within UML nfs-mount the exported directory /mnt/ramdisk/victims/ onto 3 different UML directories /mnt/nfsv[234] - just to test all 3 NFS versions
8. run trinity within the UML guest using a victims directory /mnt/nfsv[234]/v1/v2 for a longer period (rather hours)
9. stop UML, Ctrl-C any running trinity / UML process
10. try to umount mnt/ramdisk/victims/
11. if that attempt fails stop the nfs service and run the umount command again - it segfaults now
12. if the 1st umount is however successfully then make a :-/
2013-05-30T19:20:28.000+02:00 n22 rpc.mountd[2921]: authenticated unmount request from 192.168.1.63:798 for /mnt/ramdisk/victims (/mnt/ramdisk/victims) 2013-05-30T19:20:28.000+02:00 n22 rpc.mountd[2921]: authenticated unmount request from 192.168.1.63:799 for /mnt/ramdisk/victims (/mnt/ramdisk/victims) 2013-05-30T19:20:42.569+02:00 n22 kernel: br0: port 1(tap0) entered disabled state 2013-05-30T19:21:10.000+02:00 n22 rpc.mountd[2921]: Caught signal 15, un-registering and exiting. 2013-05-30T19:21:10.336+02:00 n22 kernel: lockd: couldn't shutdown host module for net c161c200! 2013-05-30T19:21:10.338+02:00 n22 kernel: nfsd: last server has exited, flushing export cache 2013-05-30T19:21:12.227+02:00 n22 kernel: EXT4-fs (loop0): sb orphan head is 32315 2013-05-30T19:21:12.227+02:00 n22 kernel: sb_info orphan list: 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32315 at e8702158: mode 102357, nlink 0, next 32173 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32173 at e773a860: mode 100406, nlink 0, next 32383 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32383 at e93bbd78: mode 102041, nlink 0, next 32233 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32233 at e7e742e0: mode 103267, nlink 0, next 32421 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32421 at e84fad10: mode 100102, nlink 0, next 32155 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32155 at e8700538: mode 100700, nlink 0, next 32230 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32230 at e77397f8: mode 102747, nlink 0, next 32313 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32313 at e8701ca8: mode 102667, nlink 0, next 32244 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32244 at e79b3670: mode 100353, nlink 0, next 32361 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32361 at e8703b20: mode 100206, nlink 0, next 32271 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32271 at e79b3b20: mode 100000, nlink 0, next 32255 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32255 at eb8ec088: mode 104657, nlink 0, next 32366 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32366 at e8701f00: mode 105711, nlink 0, next 32281 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32281 at e77382e0: mode 101637, nlink 0, next 32151 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32151 at e92cce98: mode 101557, nlink 0, next 32138 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32138 at e932a608: mode 101327, nlink 0, next 32013 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32013 at e74be158: mode 101527, nlink 0, next 32012 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32012 at e74be3b0: mode 102427, nlink 0, next 32110 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32110 at e74bdf00: mode 101303, nlink 0, next 32112 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32112 at e74beab8: mode 100000, nlink 0, next 32066 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32066 at e79f9a50: mode 104607, nlink 0, next 32148 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32148 at e9331ca8: mode 102507, nlink 0, next 32158 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32158 at e84c31c0: mode 100000, nlink 0, next 32139 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32139 at e84c1ca8: mode 101507, nlink 0, next 32115 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32115 at e93310f0: mode 104037, nlink 0, next 0 2013-05-30T19:21:12.228+02:00 n22 kernel: ------------[ cut here ]------------ 2013-05-30T19:21:12.228+02:00 n22 kernel: kernel BUG at fs/ext4/super.c:804! 2013-05-30T19:21:12.228+02:00 n22 kernel: invalid opcode: 0000 [#1] SMP 2013-05-30T19:21:12.228+02:00 n22 kernel: Modules linked in: loop nfsd auth_rpcgss oid_registry lockd sunrpc ip6t_REJECT ip6table_filter ip6_tables ipt_MASQUERADE xt_owner xt_LOG xt_limit xt_multiport ipt_REJECT xt_tcpudp xt_recent xt_conntrack iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables x_tables af_packet pppoe pppox ppp_generic slhc bridge stp llc ipv6 tun fuse dm_mod coretemp kvm_intel kvm aesni_intel i915 xts aes_i586 lrw gf128mul ablk_helper arc4 hid_cherry hid_generic iwldvm fbcon snd_hda_codec_conexant cfbfillrect cfbimgblt cryptd i2c_algo_bit sr_mod cfbcopyarea intel_agp sdhci_pci cdrom intel_gtt evdev mac80211 sdhci bitblit mmc_core softcursor font acpi_cpufreq mperf psmouse usbhid drm_kms_helper usblp snd_hda_intel e1000e uvcvideo drm videobuf2_vmalloc hid agpgart videobuf2_memops videobuf2_core videodev fb 8250_pci snd_hda_codec ptp i2c_i801 8250 pps_core processor battery fbdev iwlwifi i2c_core cfg80211 thermal wmi tpm_tis snd_pcm snd_page_alloc snd_timer tpm tpm_bios thinkpad_acpi video nvram snd soundcore ac rfkill thermal_sys button serial_core hwmon [last unloaded: microcode] 2013-05-30T19:21:12.228+02:00 n22 kernel: CPU: 1 PID: 11831 Comm: umount Not tainted 3.10.0-rc3+ #6 2013-05-30T19:21:12.228+02:00 n22 kernel: Hardware name: LENOVO 4180F65/4180F65, BIOS 83ET73WW (1.43 ) 11/30/2012 2013-05-30T19:21:12.228+02:00 n22 kernel: task: eec69aa0 ti: eb4b6000 task.ti: eb4b6000 2013-05-30T19:21:12.228+02:00 n22 kernel: EIP: 0060:[<c11ba6ec>] EFLAGS: 00010287 CPU: 1 2013-05-30T19:21:12.228+02:00 n22 kernel: EIP is at ext4_put_super+0x2dc/0x2e0 2013-05-30T19:21:12.228+02:00 n22 kernel: EAX: 0000003d EBX: eaa3d400 ECX: eaa3d550 EDX: eaa3d550 2013-05-30T19:21:12.228+02:00 n22 kernel: ESI: eaa3f000 EDI: eaa3d514 EBP: eb4b7efc ESP: eb4b7ecc 2013-05-30T19:21:12.228+02:00 n22 kernel: DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 2013-05-30T19:21:12.228+02:00 n22 kernel: CR0: 80050033 CR2: b6bab000 CR3: 2edc6000 CR4: 000407f0 2013-05-30T19:21:12.229+02:00 n22 kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 2013-05-30T19:21:12.229+02:00 n22 kernel: DR6: ffff0ff0 DR7: 00000400 2013-05-30T19:21:12.229+02:00 n22 kernel: Stack: 2013-05-30T19:21:12.229+02:00 n22 kernel: c1567fa0 eaa3f1bc 00007d73 e93310f0 0000881f 00000000 00000000 e93310d0 2013-05-30T19:21:12.229+02:00 n22 kernel: eaa3d550 eaa3f000 eaa3f058 c14a06e0 eb4b7f18 c111f771 eb4b7f28 eb4b7f18 2013-05-30T19:21:12.229+02:00 n22 kernel: f1d70400 00000083 eaa3f000 eb4b7f28 c111f819 eaa3f000 c15fde28 eb4b7f38 2013-05-30T19:21:12.229+02:00 n22 kernel: Call Trace: 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c111f771>] generic_shutdown_super+0x51/0xd0 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c111f819>] kill_block_super+0x29/0x70 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c111fa64>] deactivate_locked_super+0x44/0x70 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c1120437>] deactivate_super+0x47/0x60 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c11371bd>] mntput_no_expire+0xcd/0x120 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c113807e>] SyS_umount+0xae/0x330 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c113831e>] SyS_oldumount+0x1e/0x20 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c1482701>] sysenter_do_call+0x12/0x22 2013-05-30T19:21:12.229+02:00 n22 kernel: Code: 24 a0 7f 56 c1 05 bc 01 00 00 89 44 24 04 e8 d2 f8 2b 00 8b 4d ec 8b 55 f0 8b 09 39 ca 75 b2 39 93 50 01 00 00 0f 84 9a fe ff ff <0f> 0b 66 90 55 89 e5 83 ec 20 66 66 66 66 90 8d 45 18 c7 04 24 2013-05-30T19:21:12.229+02:00 n22 kernel: EIP: [<c11ba6ec>] ext4_put_super+0x2dc/0x2e0 SS:ESP 0068:eb4b7ecc 2013-05-30T19:21:12.229+02:00 n22 kernel: ---[ end trace 2a52a524ae176def ]---
Signature · 7 lines
MfG/Sincerely Toralf Förster pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3 -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 1 line · Toralf Förster
11. if that attempt fails stop the nfs service and run the umount command again - it segfaults now
forgot to mention that after that just sysrq+b thelps me to reboot the machine, a "sync" does not come back nor can be interrupted with Ctrl-C, "reboot" don't work too - OTOH I'm able to write an email and save it as a draft.
Signature · 7 lines
MfG/Sincerely Toralf Förster pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3 -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines · Toralf Förster
Quoted 1 line · Toralf Förster
11. if that attempt fails stop the nfs service and run the umount command again - it segfaults now
forgot to mention that after that just sysrq+b thelps me to reboot the machine, a "sync" does not come back nor can be interrupted with Ctrl-C, "reboot" don't work too - OTOH I'm able to write an email and save it as a draft.
sysrq+w gave :
2013-06-01T15:30:25.825+02:00 n22 kernel: SysRq : Show Blocked State 2013-06-01T15:30:25.825+02:00 n22 kernel: task PC stack pid father 2013-06-01T15:30:25.825+02:00 n22 kernel: kworker/0:1 D f08f7db0 0 29 2 0x00000000 2013-06-01T15:30:25.825+02:00 n22 kernel: Workqueue: events do_sync_work 2013-06-01T15:30:25.825+02:00 n22 kernel: f22b9e88 00000046 00000000 f08f7db0 00000008 e7a9080b 00000a16 f22b9e34 2013-06-01T15:30:25.825+02:00 n22 kernel: c16b1400 c104ec08 c16b1400 f37ba400 f21b8d50 ef1a39b0 00000296 b9b6e892 2013-06-01T15:30:25.825+02:00 n22 kernel: f22b9eb0 7fffffff f22b9eac f22b9e78 c14803f3 f21b8d50 00000001 f21b8d50 2013-06-01T15:30:25.825+02:00 n22 kernel: Call Trace: 2013-06-01T15:30:25.825+02:00 n22 kernel: [<c104ec08>] ? try_to_grab_pending+0x98/0x130 2013-06-01T15:30:25.825+02:00 n22 kernel: [<c14803f3>] ? wait_for_completion+0x83/0xc0 2013-06-01T15:30:25.826+02:00 n22 kernel: [<c1062410>] ? try_to_wake_up+0x220/0x220 2013-06-01T15:30:25.826+02:00 n22 kernel: [<c1480453>] schedule+0x23/0x60 2013-06-01T15:30:25.826+02:00 n22 kernel: [<c1480d15>] rwsem_down_read_failed+0x75/0xb2 2013-06-01T15:30:25.826+02:00 n22 kernel: [<c12da18f>] call_rwsem_down_read_failed+0x7/0xc 2013-06-01T15:30:25.826+02:00 n22 kernel: [<c147f24c>] ? down_read+0x1c/0x1f 2013-06-01T15:30:25.826+02:00 n22 kernel: [<c11208d2>] iterate_supers+0x62/0xc0 2013-06-01T15:30:25.826+02:00 n22 kernel: [<c1144fa0>] ? fdatawrite_one_bdev+0x20/0x20 2013-06-01T15:30:25.826+02:00 n22 kernel: [<c1144f22>] do_sync_work+0x22/0x80 2013-06-01T15:30:25.826+02:00 n22 kernel: [<c104e62e>] process_one_work+0x10e/0x390 2013-06-01T15:30:25.826+02:00 n22 kernel: [<c104f602>] worker_thread+0x102/0x320 2013-06-01T15:30:25.827+02:00 n22 kernel: [<c104f500>] ? manage_workers.isra.24+0x290/0x290 2013-06-01T15:30:25.827+02:00 n22 kernel: [<c1054b74>] kthread+0x94/0xa0 2013-06-01T15:30:25.827+02:00 n22 kernel: [<c1481a37>] ret_from_kernel_thread+0x1b/0x28 2013-06-01T15:30:25.827+02:00 n22 kernel: [<c1054ae0>] ? flush_kthread_worker+0x90/0x90 2013-06-01T15:30:25.827+02:00 n22 kernel: sync D f08f7db0 0 27190 27033 0x00000000 2013-06-01T15:30:25.827+02:00 n22 kernel: ef57ff38 00000082 00000000 f08f7db0 00000008 c7f4f883 00000a15 ef57fee4 2013-06-01T15:30:25.827+02:00 n22 kernel: c16b1400 c104ec08 c16b1400 f37c7400 eeed4fe0 ef1a39b0 00000296 2a2f912a 2013-06-01T15:30:25.827+02:00 n22 kernel: ef57ff60 7fffffff ef57ff5c ef57ff28 c14803f3 eeed4fe0 00000001 eeed4fe0 2013-06-01T15:30:25.827+02:00 n22 kernel: Call Trace: 2013-06-01T15:30:25.827+02:00 n22 kernel: [<c104ec08>] ? try_to_grab_pending+0x98/0x130 2013-06-01T15:30:25.827+02:00 n22 kernel: [<c14803f3>] ? wait_for_completion+0x83/0xc0 2013-06-01T15:30:25.827+02:00 n22 kernel: [<c1062410>] ? try_to_wake_up+0x220/0x220 2013-06-01T15:30:25.827+02:00 n22 kernel: [<c1480453>] schedule+0x23/0x60 2013-06-01T15:30:25.827+02:00 n22 kernel: [<c1480d15>] rwsem_down_read_failed+0x75/0xb2 2013-06-01T15:30:25.827+02:00 n22 kernel: [<c12da18f>] call_rwsem_down_read_failed+0x7/0xc 2013-06-01T15:30:25.827+02:00 n22 kernel: [<c147f24c>] ? down_read+0x1c/0x1f 2013-06-01T15:30:25.827+02:00 n22 kernel: [<c11208d2>] iterate_supers+0x62/0xc0 2013-06-01T15:30:25.827+02:00 n22 kernel: [<c1144fa0>] ? fdatawrite_one_bdev+0x20/0x20 2013-06-01T15:30:25.827+02:00 n22 kernel: [<c11450d1>] sys_sync+0x31/0x80 2013-06-01T15:30:25.827+02:00 n22 kernel: [<c1481ac1>] sysenter_do_call+0x12/0x22 2013-06-01T15:30:25.828+02:00 n22 kernel: kworker/0:0 D f08f7db0 0 27191 2 0x00000000 2013-06-01T15:30:25.828+02:00 n22 kernel: Workqueue: events do_sync_work 2013-06-01T15:30:25.828+02:00 n22 kernel: ea5e9e88 00000046 00000000 f08f7db0 00000008 42845398 00000a17 eeed0336 2013-06-01T15:30:25.828+02:00 n22 kernel: c16b1400 c104ec08 c16b1400 f37ba400 eeed4290 eeed0000 00000296 c1535c4d 2013-06-01T15:30:25.828+02:00 n22 kernel: ea5e9eb0 7fffffff ea5e9eac ea5e9e78 c14803f3 eeed4290 00000001 eeed4290 2013-06-01T15:30:25.828+02:00 n22 kernel: Call Trace: 2013-06-01T15:30:25.828+02:00 n22 kernel: [<c104ec08>] ? try_to_grab_pending+0x98/0x130 2013-06-01T15:30:25.828+02:00 n22 kernel: [<c14803f3>] ? wait_for_completion+0x83/0xc0 2013-06-01T15:30:25.828+02:00 n22 kernel: [<c1062410>] ? try_to_wake_up+0x220/0x220 2013-06-01T15:30:25.828+02:00 n22 kernel: [<c1480453>] schedule+0x23/0x60 2013-06-01T15:30:25.829+02:00 n22 kernel: [<c1480d15>] rwsem_down_read_failed+0x75/0xb2 2013-06-01T15:30:25.829+02:00 n22 kernel: [<c12da18f>] call_rwsem_down_read_failed+0x7/0xc 2013-06-01T15:30:25.829+02:00 n22 kernel: [<c147f24c>] ? down_read+0x1c/0x1f 2013-06-01T15:30:25.829+02:00 n22 kernel: [<c11208d2>] iterate_supers+0x62/0xc0 2013-06-01T15:30:25.829+02:00 n22 kernel: [<c1144fa0>] ? fdatawrite_one_bdev+0x20/0x20 2013-06-01T15:30:25.829+02:00 n22 kernel: [<c1144f22>] do_sync_work+0x22/0x80 2013-06-01T15:30:25.829+02:00 n22 kernel: [<c104e62e>] process_one_work+0x10e/0x390 2013-06-01T15:30:25.829+02:00 n22 kernel: [<c104f602>] worker_thread+0x102/0x320 2013-06-01T15:30:25.829+02:00 n22 kernel: [<c104f500>] ? manage_workers.isra.24+0x290/0x290 2013-06-01T15:30:25.829+02:00 n22 kernel: [<c1054b74>] kthread+0x94/0xa0 2013-06-01T15:30:25.829+02:00 n22 kernel: [<c1481a37>] ret_from_kernel_thread+0x1b/0x28 2013-06-01T15:30:25.829+02:00 n22 kernel: [<c1054ae0>] ? flush_kthread_worker+0x90/0x90 2013-06-01T15:30:58.399+02:00 n22 kernel: SysRq : Emergency Sync
Signature · 7 lines
MfG/Sincerely Toralf Förster pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3 -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 10 lines · Toralf Förster
With kernel 3.10-rcX there's a big likelihood to observe that issue if I do the following steps:
1. create a 257 MB file
/mnt/ramdisk/disk0
2. create an EXT4 fs onto it
3. mount it onto/mnt/ramdisk/victims/
4. create files and directories in/mnt/ramdisk/victims/v1/v2
5. exportfs the directory/mnt/ramdisk/victims/via NFS
6. start a user mode linux
7. within UML nfs-mount the exported directory/mnt/ramdisk/victims/onto 3 different UML directories /mnt/nfsv[234] - just to test all 3 NFS versions
8. run trinity within the UML guest using a victims directory /mnt/nfsv[234]/v1/v2 for a longer period (rather hours)
And therein lies the unknown magic.
Again, trinity's job is to try to corrupt the kernel by fuzzing syscalls. We've had "xfs bug reports" after running trinity as well... and all indications are that xfs is the victim, not the root cause.
It could be a filesystem bug, or just as easily some other bug in a syscall that allowed trinity to corrupt memory.
I do not think these bug reports are actionable until you can figure out how to narrow down the trinity operations that cause the problem.
-Eric
Quoted 34 lines
9. stop UML, Ctrl-C any running trinity / UML process
10. try to umount mnt/ramdisk/victims/
11. if that attempt fails stop the nfs service and run the umount command again - it segfaults now
12. if the 1st umount is however successfully then make a :-/2013-05-30T19:20:28.000+02:00 n22 rpc.mountd[2921]: authenticated unmount request from 192.168.1.63:798 for /mnt/ramdisk/victims (/mnt/ramdisk/victims) 2013-05-30T19:20:28.000+02:00 n22 rpc.mountd[2921]: authenticated unmount request from 192.168.1.63:799 for /mnt/ramdisk/victims (/mnt/ramdisk/victims) 2013-05-30T19:20:42.569+02:00 n22 kernel: br0: port 1(tap0) entered disabled state 2013-05-30T19:21:10.000+02:00 n22 rpc.mountd[2921]: Caught signal 15, un-registering and exiting. 2013-05-30T19:21:10.336+02:00 n22 kernel: lockd: couldn't shutdown host module for net c161c200! 2013-05-30T19:21:10.338+02:00 n22 kernel: nfsd: last server has exited, flushing export cache 2013-05-30T19:21:12.227+02:00 n22 kernel: EXT4-fs (loop0): sb orphan head is 32315 2013-05-30T19:21:12.227+02:00 n22 kernel: sb_info orphan list: 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32315 at e8702158: mode 102357, nlink 0, next 32173 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32173 at e773a860: mode 100406, nlink 0, next 32383 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32383 at e93bbd78: mode 102041, nlink 0, next 32233 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32233 at e7e742e0: mode 103267, nlink 0, next 32421 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32421 at e84fad10: mode 100102, nlink 0, next 32155 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32155 at e8700538: mode 100700, nlink 0, next 32230 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32230 at e77397f8: mode 102747, nlink 0, next 32313 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32313 at e8701ca8: mode 102667, nlink 0, next 32244 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32244 at e79b3670: mode 100353, nlink 0, next 32361 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32361 at e8703b20: mode 100206, nlink 0, next 32271 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32271 at e79b3b20: mode 100000, nlink 0, next 32255 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32255 at eb8ec088: mode 104657, nlink 0, next 32366 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32366 at e8701f00: mode 105711, nlink 0, next 32281 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32281 at e77382e0: mode 101637, nlink 0, next 32151 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32151 at e92cce98: mode 101557, nlink 0, next 32138 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32138 at e932a608: mode 101327, nlink 0, next 32013 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32013 at e74be158: mode 101527, nlink 0, next 32012 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32012 at e74be3b0: mode 102427, nlink 0, next 32110 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32110 at e74bdf00: mode 101303, nlink 0, next 32112 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32112 at e74beab8: mode 100000, nlink 0, next 32066
Quoted 8 lines
2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32066 at e79f9a50: mode 104607, nlink 0, next 32148 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32148 at e9331ca8: mode 102507, nlink 0, next 32158 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32158 at e84c31c0: mode 100000, nlink 0, next 32139 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32139 at e84c1ca8: mode 101507, nlink 0, next 32115 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32115 at e93310f0: mode 104037, nlink 0, next 0 2013-05-30T19:21:12.228+02:00 n22 kernel: ------------[ cut here ]------------ 2013-05-30T19:21:12.228+02:00 n22 kernel: kernel BUG at fs/ext4/super.c:804! 2013-05-30T19:21:12.228+02:00 n22 kernel: invalid opcode: 0000 [#1] SMP 2013-05-30T19:21:12.228+02:00 n22 kernel: Modules linked in: loop nfsd auth_rpcgss oid_registry lockd sunrpc ip6t_REJECT ip6table_filter ip6_tables ipt_MASQUERADE xt_owner xt_LOG xt_limit xt_multiport ipt_REJECT xt_tcpudp xt_recent xt_conntrack iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables x_tables af_packet pppoe pppox ppp_generic slhc bridge stp llc ipv6 tun fuse dm_mod coretemp kvm_intel kvm aesni_intel i915 xts aes_i586 lrw gf128mul ablk_helper arc4 hid_cherry hid_generic iwldvm fbcon snd_hda_codec_conexant cfbfillrect cfbimgblt cryptd i2c_algo_bit sr_mod cfbcopyarea intel_agp sdhci_pci cdrom intel_gtt evdev mac80211 sdhci bitblit mmc_core softcursor font acpi_cpufreq mperf psmouse usbhid drm_kms_helper usblp snd_hda_intel e1000e uvcvideo drm videobuf2_vmalloc hid agpgart videobuf2_memops videobuf2_core videodev fb 8250_pci snd_hda_codec ptp i!
2c!
Quoted 21 lines
_i8018250
pps_coreprocessor battery fbdev iwlwifii2c_corecfg80211 thermal wmitpm_tissnd_pcmsnd_page_allocsnd_timertpmtpm_biosthinkpad_acpivideo nvram snd soundcore ac rfkillthermal_sysbuttonserial_corehwmon [last unloaded: microcode]2013-05-30T19:21:12.228+02:00 n22 kernel: CPU: 1 PID: 11831 Comm: umount Not tainted 3.10.0-rc3+ #6 2013-05-30T19:21:12.228+02:00 n22 kernel: Hardware name: LENOVO 4180F65/4180F65, BIOS 83ET73WW (1.43 ) 11/30/2012 2013-05-30T19:21:12.228+02:00 n22 kernel: task: eec69aa0 ti: eb4b6000 task.ti: eb4b6000 2013-05-30T19:21:12.228+02:00 n22 kernel: EIP: 0060:[<c11ba6ec>] EFLAGS: 00010287 CPU: 1 2013-05-30T19:21:12.228+02:00 n22 kernel: EIP is at ext4_put_super+0x2dc/0x2e0 2013-05-30T19:21:12.228+02:00 n22 kernel: EAX: 0000003d EBX: eaa3d400 ECX: eaa3d550 EDX: eaa3d550 2013-05-30T19:21:12.228+02:00 n22 kernel: ESI: eaa3f000 EDI: eaa3d514 EBP: eb4b7efc ESP: eb4b7ecc 2013-05-30T19:21:12.228+02:00 n22 kernel: DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 2013-05-30T19:21:12.228+02:00 n22 kernel: CR0: 80050033 CR2: b6bab000 CR3: 2edc6000 CR4: 000407f0 2013-05-30T19:21:12.229+02:00 n22 kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 2013-05-30T19:21:12.229+02:00 n22 kernel: DR6: ffff0ff0 DR7: 00000400 2013-05-30T19:21:12.229+02:00 n22 kernel: Stack: 2013-05-30T19:21:12.229+02:00 n22 kernel: c1567fa0 eaa3f1bc 00007d73 e93310f0 0000881f 00000000 00000000 e93310d0 2013-05-30T19:21:12.229+02:00 n22 kernel: eaa3d550 eaa3f000 eaa3f058 c14a06e0 eb4b7f18 c111f771 eb4b7f28 eb4b7f18 2013-05-30T19:21:12.229+02:00 n22 kernel: f1d70400 00000083 eaa3f000 eb4b7f28 c111f819 eaa3f000 c15fde28 eb4b7f38 2013-05-30T19:21:12.229+02:00 n22 kernel: Call Trace: 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c111f771>] generic_shutdown_super+0x51/0xd0 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c111f819>] kill_block_super+0x29/0x70 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c111fa64>] deactivate_locked_super+0x44/0x70
Quoted 10 lines
2013-05-30T19:21:12.229+02:00 n22 kernel: [<c1120437>] deactivate_super+0x47/0x60 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c11371bd>] mntput_no_expire+0xcd/0x120 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c113807e>] SyS_umount+0xae/0x330 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c113831e>] SyS_oldumount+0x1e/0x20 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c1482701>] sysenter_do_call+0x12/0x22 2013-05-30T19:21:12.229+02:00 n22 kernel: Code: 24 a0 7f 56 c1 05 bc 01 00 00 89 44 24 04 e8 d2 f8 2b 00 8b 4d ec 8b 55 f0 8b 09 39 ca 75 b2 39 93 50 01 00 00 0f 84 9a fe ff ff <0f> 0b 66 90 55 89 e5 83 ec 20 66 66 66 66 90 8d 45 18 c7 04 24 2013-05-30T19:21:12.229+02:00 n22 kernel: EIP: [<c11ba6ec>] ext4_put_super+0x2dc/0x2e0 SS:ESP 0068:eb4b7ecc 2013-05-30T19:21:12.229+02:00 n22 kernel: ---[ end trace 2a52a524ae176def ]---
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines · Eric Sandeen
And therein lies the unknown magic.
Again, trinity's job is to try to corrupt the kernel by fuzzing syscalls. We've had "xfs bug reports" after running trinity as well... and all indications are that xfs is the victim, not the root cause.
It could be a filesystem bug, or just as easily some other bug in a syscall that allowed trinity to corrupt memory.
I do not think these bug reports are actionable until you can figure out how to narrow down the trinity operations that cause the problem.
ok, I'm really trying to get a scenario without trinity.
I'm convinced that such a scenario does exist, just because kernel 3.9.4 does not run into those issue whereas 3.10 does.
Signature · 7 lines
MfG/Sincerely Toralf Förster pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3 -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 3 lines · Eric Sandeen
And therein lies the unknown magic.
Again, trinity's job is to try to corrupt the kernel by fuzzing syscalls
Just one question :
Is trinity able to corrupt the host kernel while running within a user mode linux guest ?
Because it is the host kernel which fails after being attacked with a lot of NFS calls.
Signature · 7 lines
MfG/Sincerely Toralf Förster pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3 -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 12 lines · Eric Sandeen
Quoted 10 lines · Toralf Förster
With kernel 3.10-rcX there's a big likelihood to observe that issue if I do the following steps:
1. create a 257 MB file
/mnt/ramdisk/disk0
2. create an EXT4 fs onto it
3. mount it onto/mnt/ramdisk/victims/
4. create files and directories in/mnt/ramdisk/victims/v1/v2
5. exportfs the directory/mnt/ramdisk/victims/via NFS
6. start a user mode linux
7. within UML nfs-mount the exported directory/mnt/ramdisk/victims/onto 3 different UML directories /mnt/nfsv[234] - just to test all 3 NFS versions
8. run trinity within the UML guest using a victims directory /mnt/nfsv[234]/v1/v2 for a longer period (rather hours)
Quoted 3 lines
And therein lies the unknown magic.
Again, trinity's job is to try to corrupt the kernel by fuzzing syscalls.
But does trintiy corrupts the host kernel if it runs within a cirtualized environemtn (ok, "just" a user mode linux image) ?
Quoted 2 lines
It could be a filesystem bug, or just as easily some other bug in a syscall that allowed trinity to corrupt memory.
I bet that it is related ot the interaction of NFS and EXT4FS, because the host is mostly stressed by scary file system calls calls coming from the client over BFS (==trinity)
Quoted 3 lines
I do not think these bug reports are actionable until you can figure out how to narrow down the trinity operations that cause the problem.
-Eric
I really try to get a scenario.
With the latest trinity versions at least the handling of the fuzz testing becomes much more easier and reliable.
FWIW with 3.10 for the host kernel that bug appears now much more often than with 3.9.X (for the host, the UML client runs mostly latest git tree + 3 UML patches)
Quoted 76 lines
Quoted 34 lines
9. stop UML, Ctrl-C any running trinity / UML process
10. try to umount mnt/ramdisk/victims/
11. if that attempt fails stop the nfs service and run the umount command again - it segfaults now
12. if the 1st umount is however successfully then make a :-/2013-05-30T19:20:28.000+02:00 n22 rpc.mountd[2921]: authenticated unmount request from 192.168.1.63:798 for /mnt/ramdisk/victims (/mnt/ramdisk/victims) 2013-05-30T19:20:28.000+02:00 n22 rpc.mountd[2921]: authenticated unmount request from 192.168.1.63:799 for /mnt/ramdisk/victims (/mnt/ramdisk/victims) 2013-05-30T19:20:42.569+02:00 n22 kernel: br0: port 1(tap0) entered disabled state 2013-05-30T19:21:10.000+02:00 n22 rpc.mountd[2921]: Caught signal 15, un-registering and exiting. 2013-05-30T19:21:10.336+02:00 n22 kernel: lockd: couldn't shutdown host module for net c161c200! 2013-05-30T19:21:10.338+02:00 n22 kernel: nfsd: last server has exited, flushing export cache 2013-05-30T19:21:12.227+02:00 n22 kernel: EXT4-fs (loop0): sb orphan head is 32315 2013-05-30T19:21:12.227+02:00 n22 kernel: sb_info orphan list: 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32315 at e8702158: mode 102357, nlink 0, next 32173 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32173 at e773a860: mode 100406, nlink 0, next 32383 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32383 at e93bbd78: mode 102041, nlink 0, next 32233 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32233 at e7e742e0: mode 103267, nlink 0, next 32421 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32421 at e84fad10: mode 100102, nlink 0, next 32155 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32155 at e8700538: mode 100700, nlink 0, next 32230 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32230 at e77397f8: mode 102747, nlink 0, next 32313 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32313 at e8701ca8: mode 102667, nlink 0, next 32244 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32244 at e79b3670: mode 100353, nlink 0, next 32361 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32361 at e8703b20: mode 100206, nlink 0, next 32271 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32271 at e79b3b20: mode 100000, nlink 0, next 32255 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32255 at eb8ec088: mode 104657, nlink 0, next 32366 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32366 at e8701f00: mode 105711, nlink 0, next 32281 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32281 at e77382e0: mode 101637, nlink 0, next 32151 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32151 at e92cce98: mode 101557, nlink 0, next 32138 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32138 at e932a608: mode 101327, nlink 0, next 32013 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32013 at e74be158: mode 101527, nlink 0, next 32012 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32012 at e74be3b0: mode 102427, nlink 0, next 32110 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32110 at e74bdf00: mode 101303, nlink 0, next 32112 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32112 at e74beab8: mode 100000, nlink 0, next 32066Quoted 8 lines
2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32066 at e79f9a50: mode 104607, nlink 0, next 32148 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32148 at e9331ca8: mode 102507, nlink 0, next 32158 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32158 at e84c31c0: mode 100000, nlink 0, next 32139 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32139 at e84c1ca8: mode 101507, nlink 0, next 32115 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32115 at e93310f0: mode 104037, nlink 0, next 0 2013-05-30T19:21:12.228+02:00 n22 kernel: ------------[ cut here ]------------ 2013-05-30T19:21:12.228+02:00 n22 kernel: kernel BUG at fs/ext4/super.c:804! 2013-05-30T19:21:12.228+02:00 n22 kernel: invalid opcode: 0000 [#1] SMP 2013-05-30T19:21:12.228+02:00 n22 kernel: Modules linked in: loop nfsd auth_rpcgss oid_registry lockd sunrpc ip6t_REJECT ip6table_filter ip6_tables ipt_MASQUERADE xt_owner xt_LOG xt_limit xt_multiport ipt_REJECT xt_tcpudp xt_recent xt_conntrack iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables x_tables af_packet pppoe pppox ppp_generic slhc bridge stp llc ipv6 tun fuse dm_mod coretemp kvm_intel kvm aesni_intel i915 xts aes_i586 lrw gf128mul ablk_helper arc4 hid_cherry hid_generic iwldvm fbcon snd_hda_codec_conexant cfbfillrect cfbimgblt cryptd i2c_algo_bit sr_mod cfbcopyarea intel_agp sdhci_pci cdrom intel_gtt evdev mac80211 sdhci bitblit mmc_core softcursor font acpi_cpufreq mperf psmouse usbhid drm_kms_helper usblp snd_hda_intel e1000e uvcvideo drm videobuf2_vmalloc hid agpgart videobuf2_memops videobuf2_core videodev fb 8250_pci snd_hda_codec ptp i!2c!
Quoted 31 lines
_i8018250
pps_coreprocessor battery fbdev iwlwifii2c_corecfg80211 thermal wmitpm_tissnd_pcmsnd_page_allocsnd_timertpmtpm_biosthinkpad_acpivideo nvram snd soundcore ac rfkillthermal_sysbuttonserial_corehwmon [last unloaded: microcode]2013-05-30T19:21:12.228+02:00 n22 kernel: CPU: 1 PID: 11831 Comm: umount Not tainted 3.10.0-rc3+ #6 2013-05-30T19:21:12.228+02:00 n22 kernel: Hardware name: LENOVO 4180F65/4180F65, BIOS 83ET73WW (1.43 ) 11/30/2012 2013-05-30T19:21:12.228+02:00 n22 kernel: task: eec69aa0 ti: eb4b6000 task.ti: eb4b6000 2013-05-30T19:21:12.228+02:00 n22 kernel: EIP: 0060:[<c11ba6ec>] EFLAGS: 00010287 CPU: 1 2013-05-30T19:21:12.228+02:00 n22 kernel: EIP is at ext4_put_super+0x2dc/0x2e0 2013-05-30T19:21:12.228+02:00 n22 kernel: EAX: 0000003d EBX: eaa3d400 ECX: eaa3d550 EDX: eaa3d550 2013-05-30T19:21:12.228+02:00 n22 kernel: ESI: eaa3f000 EDI: eaa3d514 EBP: eb4b7efc ESP: eb4b7ecc 2013-05-30T19:21:12.228+02:00 n22 kernel: DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 2013-05-30T19:21:12.228+02:00 n22 kernel: CR0: 80050033 CR2: b6bab000 CR3: 2edc6000 CR4: 000407f0 2013-05-30T19:21:12.229+02:00 n22 kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 2013-05-30T19:21:12.229+02:00 n22 kernel: DR6: ffff0ff0 DR7: 00000400 2013-05-30T19:21:12.229+02:00 n22 kernel: Stack: 2013-05-30T19:21:12.229+02:00 n22 kernel: c1567fa0 eaa3f1bc 00007d73 e93310f0 0000881f 00000000 00000000 e93310d0 2013-05-30T19:21:12.229+02:00 n22 kernel: eaa3d550 eaa3f000 eaa3f058 c14a06e0 eb4b7f18 c111f771 eb4b7f28 eb4b7f18 2013-05-30T19:21:12.229+02:00 n22 kernel: f1d70400 00000083 eaa3f000 eb4b7f28 c111f819 eaa3f000 c15fde28 eb4b7f38 2013-05-30T19:21:12.229+02:00 n22 kernel: Call Trace: 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c111f771>] generic_shutdown_super+0x51/0xd0 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c111f819>] kill_block_super+0x29/0x70 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c111fa64>] deactivate_locked_super+0x44/0x70 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c1120437>] deactivate_super+0x47/0x60 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c11371bd>] mntput_no_expire+0xcd/0x120 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c113807e>] SyS_umount+0xae/0x330 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c113831e>] SyS_oldumount+0x1e/0x20 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c1482701>] sysenter_do_call+0x12/0x22 2013-05-30T19:21:12.229+02:00 n22 kernel: Code: 24 a0 7f 56 c1 05 bc 01 00 00 89 44 24 04 e8 d2 f8 2b 00 8b 4d ec 8b 55 f0 8b 09 39 ca 75 b2 39 93 50 01 00 00 0f 84 9a fe ff ff <0f> 0b 66 90 55 89 e5 83 ec 20 66 66 66 66 90 8d 45 18 c7 04 24 2013-05-30T19:21:12.229+02:00 n22 kernel: EIP: [<c11ba6ec>] ext4_put_super+0x2dc/0x2e0 SS:ESP 0068:eb4b7ecc 2013-05-30T19:21:12.229+02:00 n22 kernel: ---[ end trace 2a52a524ae176def ]---
Signature · 7 lines
MfG/Sincerely Toralf Förster pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3 -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 14 lines · Eric Sandeen
Quoted 10 lines · Toralf Förster
With kernel 3.10-rcX there's a big likelihood to observe that issue if I do the following steps:
1. create a 257 MB file
/mnt/ramdisk/disk0
2. create an EXT4 fs onto it
3. mount it onto/mnt/ramdisk/victims/
4. create files and directories in/mnt/ramdisk/victims/v1/v2
5. exportfs the directory/mnt/ramdisk/victims/via NFS
6. start a user mode linux
7. within UML nfs-mount the exported directory/mnt/ramdisk/victims/onto 3 different UML directories /mnt/nfsv[234] - just to test all 3 NFS versions
8. run trinity within the UML guest using a victims directory /mnt/nfsv[234]/v1/v2 for a longer period (rather hours)And therein lies the unknown magic.
Quoted 7 lines
Again, trinity's job is to try to corrupt the kernel by fuzzing syscalls. We've had "xfs bug reports" after running trinity as well... and all indications are that xfs is the victim, not the root cause.
It could be a filesystem bug, or just as easily some other bug in a syscall that allowed trinity to corrupt memory.
I do not think these bug reports are actionable until you can figure out how to narrow down the trinity operations that cause the problem.
-Eric
Hhm,
whilst I'm not able to narrow it down to a certain trinity syscall - I can narrow it down to EXT3/EXT4 which have to be created onto a file and loop mounted to local file system and then exported via NFS at a NFS server
I can reproduce the issue using 2 user-mode-linux images within ~ 1 hour (not 100% but very often after 1 hour of fuzzying).
Trinity runs at the NFS client as a unprivileged user. It hammers the NFS server with fuzzy NFS calls. This let the NFS server image crash as soon as it then tries to unmount the NFS share.
/me wonders whether a bisect would help - assuming that it is bisectible issue.
What I get from the NFS server (UML image of a 32 bit stable Gentoo Linux) is however not too much :
Kernel panic - not syncing: BUG!
CPU: 0 PID: 1441 Comm: umount Not tainted 3.11.0-rc3-00288-gabe0308-dirty #17
652a7d68 652a7d94 08400940 084a5f7c 085d6ce0 084977e5 652a7da0 00000000
66342390 650e0f50 66342450 652a7dd0 08168632 084977e5 084ac7f4 000001c5
0841eb4c 0000182c 65e18254 000081ff 00000000 00000000 66342450 650e0f50 652a7d3c: [<0805fb1f>] show_stack+0xcf/0x100
652a7d60: [<08403897>] dump_stack+0x26/0x28
652a7d70: [<08400940>] panic+0x7a/0x18b
652a7d98: [<08168632>] ext3_put_super+0x1b2/0x240
652a7dd4: [<08101092>] generic_shutdown_super+0x52/0xc0
652a7df0: [<0810205a>] kill_block_super+0x2a/0x80
652a7e08: [<08100f2a>] deactivate_locked_super+0x2a/0x70
652a7e1c: [<08100fc1>] deactivate_super+0x51/0x70
652a7e30: [<08118dec>] mntput_no_expire+0xdc/0xf0
652a7e4c: [<0811a2d5>] SyS_umount+0x325/0x380
652a7e9c: [<0811a349>] SyS_oldumount+0x19/0x20
652a7eac: [<080618e2>] handle_syscall+0x82/0xb0
652a7ef4: [<08073c0d>] userspace+0x46d/0x590
652a7fec: [<0805e65c>] fork_handler+0x6c/0x70
652a7ffc: [<5a5a5a5a>] 0x5a5a5a5aEIP: 0073:[<40001282>] CPU: 0 Not tainted ESP: 007b:bfe44348 EFLAGS: 00000296
Not tainted
EAX: ffffffda EBX: 0804f980 ECX: 00000000 EDX: 40064ff4
ESI: 0804f878 EDI: 0804f980 EBP: 40066688 DS: 007b ES: 007b
652a7d0c: [<0807802f>] show_regs+0x10f/0x120 652a7d28: [<0806138c>] panic_exit+0x2c/0x50 652a7d38: [<0809a388>] notifier_call_chain+0x38/0x60 652a7d60: [<0809a4d3>] atomic_notifier_call_chain+0x23/0x30 652a7d70: [<08400968>] panic+0xa2/0x18b 652a7d98: [<08168632>] ext3_put_super+0x1b2/0x240 652a7dd4: [<08101092>] generic_shutdown_super+0x52/0xc0 652a7df0: [<0810205a>] kill_block_super+0x2a/0x80 652a7e08: [<08100f2a>] deactivate_locked_super+0x2a/0x70 652a7e1c: [<08100fc1>] deactivate_super+0x51/0x70 652a7e30: [<08118dec>] mntput_no_expire+0xdc/0xf0 652a7e4c: [<0811a2d5>] SyS_umount+0x325/0x380 652a7e9c: [<0811a349>] SyS_oldumount+0x19/0x20 652a7eac: [<080618e2>] handle_syscall+0x82/0xb0 652a7ef4: [<08073c0d>] userspace+0x46d/0x590 652a7fec: [<0805e65c>] fork_handler+0x6c/0x70 652a7ffc: [<5a5a5a5a>] 0x5a5a5a5a
Terminated
Quoted 76 lines
Quoted 34 lines
9. stop UML, Ctrl-C any running trinity / UML process
10. try to umount mnt/ramdisk/victims/
11. if that attempt fails stop the nfs service and run the umount command again - it segfaults now
12. if the 1st umount is however successfully then make a :-/2013-05-30T19:20:28.000+02:00 n22 rpc.mountd[2921]: authenticated unmount request from 192.168.1.63:798 for /mnt/ramdisk/victims (/mnt/ramdisk/victims) 2013-05-30T19:20:28.000+02:00 n22 rpc.mountd[2921]: authenticated unmount request from 192.168.1.63:799 for /mnt/ramdisk/victims (/mnt/ramdisk/victims) 2013-05-30T19:20:42.569+02:00 n22 kernel: br0: port 1(tap0) entered disabled state 2013-05-30T19:21:10.000+02:00 n22 rpc.mountd[2921]: Caught signal 15, un-registering and exiting. 2013-05-30T19:21:10.336+02:00 n22 kernel: lockd: couldn't shutdown host module for net c161c200! 2013-05-30T19:21:10.338+02:00 n22 kernel: nfsd: last server has exited, flushing export cache 2013-05-30T19:21:12.227+02:00 n22 kernel: EXT4-fs (loop0): sb orphan head is 32315 2013-05-30T19:21:12.227+02:00 n22 kernel: sb_info orphan list: 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32315 at e8702158: mode 102357, nlink 0, next 32173 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32173 at e773a860: mode 100406, nlink 0, next 32383 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32383 at e93bbd78: mode 102041, nlink 0, next 32233 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32233 at e7e742e0: mode 103267, nlink 0, next 32421 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32421 at e84fad10: mode 100102, nlink 0, next 32155 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32155 at e8700538: mode 100700, nlink 0, next 32230 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32230 at e77397f8: mode 102747, nlink 0, next 32313 2013-05-30T19:21:12.227+02:00 n22 kernel: inode loop0:32313 at e8701ca8: mode 102667, nlink 0, next 32244 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32244 at e79b3670: mode 100353, nlink 0, next 32361 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32361 at e8703b20: mode 100206, nlink 0, next 32271 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32271 at e79b3b20: mode 100000, nlink 0, next 32255 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32255 at eb8ec088: mode 104657, nlink 0, next 32366 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32366 at e8701f00: mode 105711, nlink 0, next 32281 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32281 at e77382e0: mode 101637, nlink 0, next 32151 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32151 at e92cce98: mode 101557, nlink 0, next 32138 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32138 at e932a608: mode 101327, nlink 0, next 32013 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32013 at e74be158: mode 101527, nlink 0, next 32012 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32012 at e74be3b0: mode 102427, nlink 0, next 32110 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32110 at e74bdf00: mode 101303, nlink 0, next 32112 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32112 at e74beab8: mode 100000, nlink 0, next 32066Quoted 8 lines
2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32066 at e79f9a50: mode 104607, nlink 0, next 32148 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32148 at e9331ca8: mode 102507, nlink 0, next 32158 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32158 at e84c31c0: mode 100000, nlink 0, next 32139 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32139 at e84c1ca8: mode 101507, nlink 0, next 32115 2013-05-30T19:21:12.228+02:00 n22 kernel: inode loop0:32115 at e93310f0: mode 104037, nlink 0, next 0 2013-05-30T19:21:12.228+02:00 n22 kernel: ------------[ cut here ]------------ 2013-05-30T19:21:12.228+02:00 n22 kernel: kernel BUG at fs/ext4/super.c:804! 2013-05-30T19:21:12.228+02:00 n22 kernel: invalid opcode: 0000 [#1] SMP 2013-05-30T19:21:12.228+02:00 n22 kernel: Modules linked in: loop nfsd auth_rpcgss oid_registry lockd sunrpc ip6t_REJECT ip6table_filter ip6_tables ipt_MASQUERADE xt_owner xt_LOG xt_limit xt_multiport ipt_REJECT xt_tcpudp xt_recent xt_conntrack iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ip_tables x_tables af_packet pppoe pppox ppp_generic slhc bridge stp llc ipv6 tun fuse dm_mod coretemp kvm_intel kvm aesni_intel i915 xts aes_i586 lrw gf128mul ablk_helper arc4 hid_cherry hid_generic iwldvm fbcon snd_hda_codec_conexant cfbfillrect cfbimgblt cryptd i2c_algo_bit sr_mod cfbcopyarea intel_agp sdhci_pci cdrom intel_gtt evdev mac80211 sdhci bitblit mmc_core softcursor font acpi_cpufreq mperf psmouse usbhid drm_kms_helper usblp snd_hda_intel e1000e uvcvideo drm videobuf2_vmalloc hid agpgart videobuf2_memops videobuf2_core videodev fb 8250_pci snd_hda_codec ptp i!2c!
Quoted 31 lines
_i8018250
pps_coreprocessor battery fbdev iwlwifii2c_corecfg80211 thermal wmitpm_tissnd_pcmsnd_page_allocsnd_timertpmtpm_biosthinkpad_acpivideo nvram snd soundcore ac rfkillthermal_sysbuttonserial_corehwmon [last unloaded: microcode]2013-05-30T19:21:12.228+02:00 n22 kernel: CPU: 1 PID: 11831 Comm: umount Not tainted 3.10.0-rc3+ #6 2013-05-30T19:21:12.228+02:00 n22 kernel: Hardware name: LENOVO 4180F65/4180F65, BIOS 83ET73WW (1.43 ) 11/30/2012 2013-05-30T19:21:12.228+02:00 n22 kernel: task: eec69aa0 ti: eb4b6000 task.ti: eb4b6000 2013-05-30T19:21:12.228+02:00 n22 kernel: EIP: 0060:[<c11ba6ec>] EFLAGS: 00010287 CPU: 1 2013-05-30T19:21:12.228+02:00 n22 kernel: EIP is at ext4_put_super+0x2dc/0x2e0 2013-05-30T19:21:12.228+02:00 n22 kernel: EAX: 0000003d EBX: eaa3d400 ECX: eaa3d550 EDX: eaa3d550 2013-05-30T19:21:12.228+02:00 n22 kernel: ESI: eaa3f000 EDI: eaa3d514 EBP: eb4b7efc ESP: eb4b7ecc 2013-05-30T19:21:12.228+02:00 n22 kernel: DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 2013-05-30T19:21:12.228+02:00 n22 kernel: CR0: 80050033 CR2: b6bab000 CR3: 2edc6000 CR4: 000407f0 2013-05-30T19:21:12.229+02:00 n22 kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 2013-05-30T19:21:12.229+02:00 n22 kernel: DR6: ffff0ff0 DR7: 00000400 2013-05-30T19:21:12.229+02:00 n22 kernel: Stack: 2013-05-30T19:21:12.229+02:00 n22 kernel: c1567fa0 eaa3f1bc 00007d73 e93310f0 0000881f 00000000 00000000 e93310d0 2013-05-30T19:21:12.229+02:00 n22 kernel: eaa3d550 eaa3f000 eaa3f058 c14a06e0 eb4b7f18 c111f771 eb4b7f28 eb4b7f18 2013-05-30T19:21:12.229+02:00 n22 kernel: f1d70400 00000083 eaa3f000 eb4b7f28 c111f819 eaa3f000 c15fde28 eb4b7f38 2013-05-30T19:21:12.229+02:00 n22 kernel: Call Trace: 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c111f771>] generic_shutdown_super+0x51/0xd0 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c111f819>] kill_block_super+0x29/0x70 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c111fa64>] deactivate_locked_super+0x44/0x70 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c1120437>] deactivate_super+0x47/0x60 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c11371bd>] mntput_no_expire+0xcd/0x120 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c113807e>] SyS_umount+0xae/0x330 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c113831e>] SyS_oldumount+0x1e/0x20 2013-05-30T19:21:12.229+02:00 n22 kernel: [<c1482701>] sysenter_do_call+0x12/0x22 2013-05-30T19:21:12.229+02:00 n22 kernel: Code: 24 a0 7f 56 c1 05 bc 01 00 00 89 44 24 04 e8 d2 f8 2b 00 8b 4d ec 8b 55 f0 8b 09 39 ca 75 b2 39 93 50 01 00 00 0f 84 9a fe ff ff <0f> 0b 66 90 55 89 e5 83 ec 20 66 66 66 66 90 8d 45 18 c7 04 24 2013-05-30T19:21:12.229+02:00 n22 kernel: EIP: [<c11ba6ec>] ext4_put_super+0x2dc/0x2e0 SS:ESP 0068:eb4b7ecc 2013-05-30T19:21:12.229+02:00 n22 kernel: ---[ end trace 2a52a524ae176def ]---
Signature · 7 lines
MfG/Sincerely Toralf Förster pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3 -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
forget to attach the back trace of the uML of an EXT3 est case :
tfoerste@n22 ~ $ gdb --core=/mnt/ramdisk/core /usr/local/bin/linux-v3.11-rc3-288-gabe0308 -n -batch -ex bt
warning: core file may not match specified executable file.
[New LWP 8384]
Core was generated by `/usr/local/bin/linux-v3.11-rc3-288-gabe0308 earlyprintk ubda=/home/tfoerste/vir'. Program terminated with signal 6, Aborted.
#0 0xb7735424 in __kernel_vsyscall () #0 0xb7735424 in __kernel_vsyscall () #1 0x083a3185 in kill () #2 0x0807163d in uml_abort () at arch/um/os-Linux/util.c:93 #3 0x08071925 in os_dump_core () at arch/um/os-Linux/util.c:138 #4 0x080613a7 in panic_exit (self=0x85a1518 <panic_exit_notifier>, unused1=0, unused2=0x85d6ce0 <buf.15904>) at arch/um/kernel/um_arch.c:240 #5 0x0809a388 in notifier_call_chain (nl=0x0, val=0, v=0x85d6ce0 <buf.15904>, nr_to_call=-2, nr_calls=0x0) at kernel/notifier.c:93 #6 0x0809a4d3 in __atomic_notifier_call_chain (nr_calls=<optimized out>, nr_to_call=<optimized out>, v=<optimized out>, val=<optimized out>, nh=<optimized out>) at kernel/notifier.c:182 #7 atomic_notifier_call_chain (nh=0x85d6cc4 <panic_notifier_list>, val=0, v=0x85d6ce0 <buf.15904>) at kernel/notifier.c:191 #8 0x08400968 in panic (fmt=0x0) at kernel/panic.c:128 #9 0x08168632 in ext3_put_super (sb=0x650e0f50) at fs/ext3/super.c:453
#10 0x08101092 in generic_shutdown_super (sb=0x650e0f50) at fs/super.c:418
#11 0x0810205a in kill_block_super (sb=0x0) at fs/super.c:1028
#12 0x08100f2a in deactivate_locked_super (s=0x650e0f50) at fs/super.c:299
#13 0x08100fc1 in deactivate_super (s=0x650e0f50) at fs/super.c:324
#14 0x08118dec in mntfree (mnt=<optimized out>) at fs/namespace.c:891
#15 mntput_no_expire (mnt=0x0) at fs/namespace.c:929
#16 0x0811a2d5 in SYSC_umount (flags=<optimized out>, name=<optimized out>) at fs/namespace.c:1335
#17 SyS_umount (name=134543744, flags=0) at fs/namespace.c:1305
#18 0x0811a349 in SYSC_oldumount (name=<optimized out>) at fs/namespace.c:1347
#19 SyS_oldumount (name=134543744) at fs/namespace.c:1345
#20 0x080618e2 in handle_syscall (r=0x663f6bd4) at arch/um/kernel/skas/syscall.c:35
#21 0x08073c0d in handle_trap (local_using_sysemu=<optimized out>, regs=<optimized out>, pid=<optimized out>) at arch/um/os-Linux/skas/process.c:198 #22 userspace (regs=0x663f6bd4) at arch/um/os-Linux/skas/process.c:431
#23 0x0805e65c in fork_handler () at arch/um/kernel/process.c:160
#24 0x5a5a5a5a in ?? ()
Signature · 3 lines
MfG/Sincerely Toralf Förster pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3